Culture state determination based on direction-dependent image information

ABSTRACT

The present disclosure provides a technique which makes it possible to evaluate a state of a cell aggregation of one or more spheroids. In the culture state determination device according to the present disclosure, a plurality of light sources sequentially illuminate a plurality of cell aggregations put on an image sensor. The image sensor acquires captured images of the plurality of the cell aggregations each time when the plurality of the light sources illuminate the plurality of the cell aggregations. Control circuitry extracts a region including an image of the cell aggregation in the captured image; generates three-dimensional image information of the region using a plurality of the captured images; extracts an outer shape of the cell aggregation and a cavity part inside the cell aggregation using the three-dimensional image information; calculates a first volume that is a volume based on the outer shape of each of the cell aggregation and a second volume that is a volume of the cavity part based on the cavity part of each of the cell aggregation in the three-dimensional image information; and determines a culture state of the cell aggregations using the first volume and the second volume.

BACKGROUND 1. Technical Field

The present disclosure relates to a culture state determinationutilizing a technique for generating an image of an object at anarbitrary focal plane.

2. Description of the Related Art

Continuous observation of cultured cells without staining is required inmany fields where the cultured cells are used in medical and industrialfields, such as production of therapeutic cells and testing of drugefficacy. As a culture method, there is a method of culturing cells as acell aggregation referred to as a spheroid. In order to determinequality of a state of a spheroid containing a large number of culturedcells, a technique for determining the state of the cultured cells basedon a captured image of the spheroid using a microscope has beenproposed.

For example, Patent Literatures 1 to 3 disclose techniques fordetermining the quality of the state of the spheroid. In PatentLiterature 1, an image of the spheroid is captured through a microscope,and the circularity and sharpness of the outer shape of the spheroid aredetermined from the acquired image, and the collapse state of thespheroid is determined from the luminance distribution of the spheroidimage. In addition, in Patent Literature 2, the quality of the state ofthe spheroid is determined from the circularity of the outline of thespheroid in the image. In addition, in Patent Literature 3, bymanipulating the gene of the cell contained in the spheroid, the cell isadjusted in such a way that a photoprotein is produced, and emits lightwithout a light source. Furthermore, the three-dimensional informationof the spheroid is synthesized from the result of the image provided bycapturing the spheroid including the above-described cells at theplurality of the focal planes using the microscope.

CITATION LIST Patent Literature

-   Patent Literature 1: WO2015/145872-   Patent Literature 2: WO2016/158719-   Patent Literature 3: WO2016/117089-   Patent Literature 4: United States Patent Application Publication    No. 2017/0192219

SUMMARY

However, since the techniques of Patent Literatures 1 and 2 evaluate thestate of the spheroid from the shape of the spheroid and the luminancedistribution on the surface of the spheroid, it is difficult to evaluatethe state of the inside of the spheroid. Although the technique ofPatent Literature 3 can evaluate the state the inside of the spheroidbased on the three-dimensional information of the spheroid, since itmanipulates the gene of the cell contained in the spheroid, it isdifficult to use the technique for a cell for treatment. In addition,the techniques of Patent Literatures 1 to 3 can determine the quality ofthe culture state of individual spheroids; however, it is difficult tochoose spheroids which are in a good culture state and usable from alarge amount of spheroids cultured for medical use or industrial use.

The present disclosure provides a culture state determination device anda culture state determination method, both of which can evaluate thestate of a cell aggregation such as one or more spheroids.

The culture state determination device of the present disclosurecomprises:

a plurality of light sources;

an image sensor on which a cell aggregation is to be mounted; and

control circuitry which, in operation,

(a) repeatedly causes the image sensor to acquire a captured imageincluding the cell aggregation when the cell aggregation is illuminatedwith each of the plurality of the light sources sequentially, to acquirea plurality of captured images;

wherein

each of the plurality of the captured images includes the cellaggregation,

(b) extracts an image region including the cell aggregation from each ofthe plurality of the captured images;

(c) generates three-dimensional image information with regard to theimage region with the plurality of the captured images;

(d) calculates a first volume and a second volume from thethree-dimensional image information;

wherein

the first volume is an entire volume of the cell aggregation; and

the second volume is a volume of a cavity part of the cell aggregation;and

(e) determines a culture state of the cell aggregation using the firstvolume and the second volume.

Another culture state determination device of the present disclosurecomprises:

a plurality of light sources;

an image sensor on which a plurality of cell aggregations are to bemounted; and

control circuitry which, in operation,

(a) repeatedly causes the image sensor to acquire a captured imageincluding at least one cell aggregation included in the plurality of thecell aggregations when the plurality of the cell aggregations areilluminated with each of the plurality of the light sourcessequentially, to acquire a plurality of captured images;

wherein

each of the plurality of the captured images includes the at least onecell aggregation included in the plurality of the cell aggregations,

(b) extracts an image region including one cell aggregation from each ofthe plurality of the captured images;

(c) generates three-dimensional image information with regard to theimage region with the plurality of the captured images;

(d) calculates a first volume and a second volume from thethree-dimensional image information;

wherein

the first volume is an entire volume of the one cell aggregation; and

the second volume is a volume of a cavity part of the one cellaggregation; and

(e) determines a culture state of the at least one cell aggregationusing the first volume and the second volume.

The method for determining a culture state of the present disclosurecomprises:

(a) repeatedly causing an image sensor to acquire a captured imageincluding a cell aggregation when the cell aggregation is illuminatedwith each of a plurality of light sources sequentially, to acquire aplurality of captured images;

wherein

each of the plurality of the captured images includes the cellaggregation,

(b) extracting an image region including the cell aggregation from eachof the plurality of the captured images;

(c) generating three-dimensional image information with regard to theimage region with the plurality of the captured images;

(d) calculating a first volume and a second volume from thethree-dimensional image information;

wherein

the first volume is an entire volume of the cell aggregation; and

the second volume is a volume of a cavity part of the cell aggregation;and

(e) determining a culture state of the cell aggregation using the firstvolume and the second volume.

Another method for determining a culture state of the present disclosurecomprises:

(a) repeatedly causing an image sensor to acquire a captured imageincluding at least one cell aggregation included in a plurality of cellaggregations when the plurality of the cell aggregations are illuminatedwith each of a plurality of light sources sequentially, to acquire aplurality of captured images;

wherein

each of the plurality of the captured images includes the at least onecell aggregation included in the plurality of the cell aggregations,

(b) extracting an image region including one cell aggregation from eachof the plurality of the captured images;

(c) generating three-dimensional image information with regard to theimage region with the plurality of the captured images;

(d) calculating a first volume and a second volume from thethree-dimensional image information;

wherein

the first volume is an entire volume of the one cell aggregation; and

the second volume is a volume of a cavity part of the one cellaggregation; and

(e) determining a culture state of the at least one cell aggregationusing the first volume and the second volume.

The comprehensive or specific aspect described above may be realized bya system, a device, a method, an integrated circuit, a computer program,or a recording medium such as a computer-readable recording disk. Thecomprehensive or specific aspect described above may be realized by anycombination of the system, the device, the method, the integratedcircuit, the computer program and the recording medium. Thecomputer-readable recording medium includes a non-volatile recordingmedium such as a compact disc-read only memory (i.e., CD-ROM).

The present disclosure allows a state of one or more cell aggregation tobe evaluated.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating one example of a functionalconfiguration of a culture state determination device according to afirst embodiment.

FIG. 2 is a block diagram illustrating one example of a functionalconfiguration of an imaging device of FIG. 1 .

FIG. 3 is a side view schematically showing one example of arelationship between a plurality of illuminators and an image sensor inthe culture state determination device according to the firstembodiment.

FIG. 4 is a diagram illustrating one example of contents stored in astorage unit according to the first embodiment.

FIG. 5 is a diagram illustrating one example of the contents stored inthe storage unit according to the first embodiment.

FIG. 6 is a block diagram illustrating one example of a functionalconfiguration of an internal image generation unit according to thefirst embodiment.

FIG. 7 is a diagram illustrating one example of contents stored in afocal plane table according to the first embodiment.

FIG. 8 is a diagram illustrating one example of the contents stored inthe storage unit according to the first embodiment.

FIG. 9A is a diagram illustrating one example of a processed image of aspheroid region.

FIG. 9B is a diagram illustrating one example of a processed image ofthe spheroid region.

FIG. 9C is a diagram illustrating one example of a processed image ofthe spheroid region.

FIG. 9D is a diagram illustrating one example of a processed image ofthe spheroid region.

FIG. 10 is a diagram illustrating one example of the contents stored inthe storage unit according to the first embodiment.

FIG. 11 is a flowchart showing one example of operation of the culturestate determination device according to the first embodiment.

FIG. 12 is a schematic perspective view illustrating one example of astate of capturing an image of the spheroid.

FIG. 13 is a diagram schematically illustrating a method of determiningthe presence or absence of a cavity part in an image provided bybinarizing an in-focus image of the spheroid.

FIG. 14A is a diagram schematically illustrating one example of arelationship between an in-focus image of a spheroid region similar tothat of FIG. 9A and a pixel value on the region.

FIG. 14B is a diagram schematically illustrating one example of therelationship between the in-focus image of the spheroid region similarto that of FIG. 9B and the pixel value on the region.

FIG. 14C is a diagram schematically illustrating one example of therelationship between the in-focus image of the spheroid region similarto that of FIG. 9C and the pixel value on the region.

FIG. 14D is a diagram schematically illustrating one example of therelationship between the in-focus image of the spheroid region similarto that of FIG. 9D and the pixel value on the region.

FIG. 15 is a flowchart illustrating one example of operation of theimaging device according to the first embodiment.

FIG. 16 is a flowchart illustrating one example of operation of theinternal image generation unit according to the first embodiment.

FIG. 17 is a schematic diagram illustrating a specific example of arefocusing processing according to the first embodiment.

FIG. 18 is a schematic diagram illustrating a specific example of therefocusing processing according to the first embodiment.

FIG. 19 is a schematic diagram illustrating a specific example of therefocusing processing according to the first embodiment.

FIG. 20 is a schematic diagram illustrating a specific example of therefocusing processing according to the first embodiment.

FIG. 21 is a block diagram illustrating one example of a functionalconfiguration of a culture state determination device according to asecond embodiment.

FIG. 22 is a schematic diagram illustrating one example of the spheroidregion extracted from a reference captured image.

FIG. 23 is a diagram illustrating one example of contents stored in astorage unit according to the second embodiment, regarding informationon the spheroid region.

FIG. 24 is a flowchart illustrating one example of operation of theculture state determination device according to the second embodiment.

FIG. 25A is a diagram illustrating one example of display by a displayunit according to the second embodiment.

FIG. 25B is a diagram illustrating one example of display by the displayunit according to the second embodiment.

FIG. 25C is a diagram illustrating one example of display by the displayunit according to the second embodiment.

FIG. 25D is a diagram illustrating one example of display by the displayunit according to the second embodiment.

DETAILED DESCRIPTION OF THE EMBODIMENT

The inventors related to the present disclosure, namely, the presentinventors have reached the following findings. When cells are culturedfor medical or industrial purposes, a large amount of spheroids areproduced simultaneously. The quality of the large amount of thespheroids is determined in a state where the large amount of thespheroids are contained in a culture vessel such as the same well as oneanother. As described in the section of “Background”, if theconventional techniques disclosed in Patent Literatures 1 to 3 are used,each spheroid is individually determined. In the above prior art, a lotof time and processing amounts are required to evaluate the quality ofall spheroids. For this reason, the present inventors have considered atechnique that makes it possible to evaluate the internal state of oneor more spheroids together. For example, the present inventors haveconsidered a technique that enables an image of one or more cellaggregation such as spheroids in the same culture vessel to besimultaneously captured, and the internal state of all the cellaggregations to be evaluated from the captured image. As a result, thepresent inventors have devised a technique as shown below.

For example, the culture state determination device of one aspect of thepresent disclosure comprises:

a plurality of light sources;

an image sensor on which a cell aggregation is to be mounted; and

control circuitry which, in operation,

(a) repeatedly causes the image sensor to acquire a captured imageincluding the cell aggregation when the cell aggregation is illuminatedwith each of the plurality of the light sources sequentially, to acquirea plurality of captured images;

wherein

each of the plurality of the captured images includes the cellaggregation,

(b) extracts an image region including the cell aggregation from each ofthe plurality of the captured images;

(c) generates three-dimensional image information with regard to theimage region with the plurality of the captured images;

(d) calculates a first volume and a second volume from thethree-dimensional image information;

wherein

the first volume is an entire volume of the cell aggregation; and

the second volume is a volume of a cavity part of the cell aggregation;and

(e) determines a culture state of the cell aggregation using the firstvolume and the second volume.

The culture state determination device of another aspect of the presentdisclosure comprises:

a plurality of light sources;

an image sensor on which a plurality of cell aggregations are to bemounted; and

control circuitry which, in operation,

(a) repeatedly causes the image sensor to acquire a captured imageincluding at least one cell aggregation included in the plurality of thecell aggregations when the plurality of the cell aggregations areilluminated with each of the plurality of the light sourcessequentially, to acquire a plurality of captured images;

wherein

each of the plurality of the captured images includes the at least onecell aggregation included in the plurality of the cell aggregations,

(b) extracts an image region including one cell aggregation from each ofthe plurality of the captured images;

(c) generates three-dimensional image information with regard to theimage region with the plurality of the captured images;

(d) calculates a first volume and a second volume from thethree-dimensional image information;

wherein

the first volume is an entire volume of the one cell aggregation; and

the second volume is a volume of a cavity part of the one cellaggregation; and

(e) determines a culture state of the at least one cell aggregationusing the first volume and the second volume.

The method for determining a culture state of one aspect of the presentdisclosure comprises:

(a) repeatedly causing an image sensor to acquire a captured imageincluding a cell aggregation when the cell aggregation is illuminatedwith each of a plurality of light sources sequentially, to acquire aplurality of captured images;

wherein

each of the plurality of the captured images includes the cellaggregation,

(b) extracting an image region including the cell aggregation from eachof the plurality of the captured images;

(c) generating three-dimensional image information with regard to theimage region with the plurality of the captured images;

(d) calculating a first volume and a second volume from thethree-dimensional image information;

wherein

the first volume is an entire volume of the cell aggregation; and

the second volume is a volume of a cavity part of the cell aggregation;and

(e) determining a culture state of the cell aggregation using the firstvolume and the second volume.

The method for determining a culture state of another aspect of thepresent disclosure comprises:

(a) repeatedly causing an image sensor to acquire a captured imageincluding at least one cell aggregation included in a plurality of cellaggregations when the plurality of the cell aggregations are illuminatedwith each of a plurality of light sources sequentially, to acquire aplurality of captured images;

wherein

each of the plurality of the captured images includes the at least onecell aggregation included in the plurality of the cell aggregations,

(b) extracting an image region including one cell aggregation from eachof the plurality of the captured images;

(c) generating three-dimensional image information with regard to theimage region with the plurality of the captured images;

(d) calculating a first volume and a second volume from thethree-dimensional image information;

wherein

the first volume is an entire volume of the one cell aggregation; and

the second volume is a volume of a cavity part of the one cellaggregation; and

(e) determining a culture state of the at least one cell aggregationusing the first volume and the second volume.

The comprehensive or specific aspect described above may be realized bya system, a device, a method, an integrated circuit, a computer program,or a recording medium such as a computer-readable recording disk. Thecomprehensive or specific aspect described above may be realized by anycombination of the system, the device, the method, the integratedcircuit, the computer program and the recording medium. Thecomputer-readable recording medium includes a non-volatile recordingmedium such as a CD-ROM. In addition, a device may be configured by oneor more devices. If a device is configured by two or more devices, thetwo or more devices may be disposed in one device or may be separatelydisposed in two or more separated devices. In the present specificationand claims, a “device” can mean not only a single device, but also asystem consisting of a plurality of devices.

Hereinafter, the culture state determination device according to thepresent disclosure will be specifically described with reference to thedrawings. Each of the embodiments described below shows a comprehensiveor specific example. Numerical values, shapes, components, arrangementpositions and connection forms of components, steps, order of the steps,and the like that will be shown in the following embodiments are merelyexamples, and are not intended to limit the present disclosure. Inaddition, among the constituent elements in the following embodiments,constituent elements that are not described in the independent claimsindicating the highest concept are described as optional constituentelements. Each drawing is a schematic diagram and is not necessarilyillustrated accurately. Furthermore, in each figure, the same referencesigns are assigned with regard to the substantially same components, andthe redundant description may be omitted or simplified.

First Embodiment

A culture state determination device 10 according to the firstembodiment will be described. FIG. 1 shows a block diagram of oneexample of a functional configuration of the culture state determinationdevice 10 according to the first embodiment. FIG. 2 is a block diagramillustrating one example of a functional configuration of an imagingdevice 100 in FIG. 1 . As shown in FIGS. 1 and 2 , the culture statedetermination device 10 according to the first embodiment illuminatessequentially a plurality of spheroids that are a plurality of objectspositioned on an image sensor 102 with a plurality of illuminators 101disposed above the image sensor 102, captures images of the plurality ofthe spheroids together using the image sensor 102 for each illumination,and acquires a plurality of captured images. Furthermore, the culturestate determination device 10 uses the acquired plurality of thecaptured images to generate images of the plurality of the spheroids inan arbitrary virtual focal plane located between the plurality of theilluminators 101 and the image sensor 102. An image on the arbitraryvirtual focal plane generated using the plurality of the captured imagesin this way is referred to as a “in-focus image”. The culture statedetermination device 10 determines the volume of the spheroid based onthe outer shape of the spheroid and the volume of the cavity part in thespheroid in the generated in-focus image, and determines the quality ofthe state of the culture of the spheroid based on the two volume ratio.The volume of the spheroid can be replaced with the number of cellscorresponding to the volume, and the volume of the cavity part can bereplaced with the number of cells corresponding to the volume. Thenumber of cells thus replaced is referred to as “pseudo cell number”.

1-1. Configuration of Culture State Determination Device According toFirst Embodiment

The configuration of the culture state determination device 10 accordingto the first embodiment will be described. As shown in FIG. 1 , theculture state determination device 10 comprises an imaging device 100, astorage unit 110, an image processing unit 120, a calculation unit 130,a state determination unit 140, and a display unit 150. Further, theimage processing unit 120 comprises a region extraction unit 121, aninternal image generation unit 122, and a discrimination unit 123.

First, the configuration of the imaging device 100 will be described. Asillustrated in FIG. 2 , the imaging device 100 comprises a plurality ofilluminators 101, an image sensor 102, and an imaging control unit 103.The imaging device 100 acquires a captured image (photographic image) ofthe object using the image sensor 102. In the present embodiment, theimaging device 100 does not have a focus lens. The imaging device 100may be formed by one device or system, may be formed by a plurality ofdevices or systems, and may be incorporated in a device or system otherthan other constituent elements of the culture state determinationdevice 10. Here, the illuminators 101 are one example of a light source.

The object is, for example, a plurality of spheroids put on the imagesensor 102. Each spheroid is a cell aggregation composed of a pluralityof translucent cells and has a three-dimensional structure. In otherwords, in the spheroid, the plurality of the cells may be positioned ina three-dimensionally overlapping way. Such a spheroid is translucentand light can pass through the spheroid. For example, the spheroid has aspherical or elliptical outer shape and has a maximum diameter of notmore than 200 μm. Here, the spheroid is an example of the cellaggregation.

Each of the plurality of the illuminators 101 outputs diffused light.The plurality of the illuminators 101 may be a plurality of illuminationdevices such as light-emitting-diodes (i.e., LEDs), may be a pluralityof light sources, and may be a plurality of light emitting elements ofdisplays. Each illuminator 101 emits light that does not intersect. Aplurality of light rays representing light emitted from one illuminatorincluded in the illuminators 101 do not cross each other. For example,regarding a first illuminator and a second illuminator, both of whichare included in the plurality of the illuminators 101, each of the firstilluminator and the second illuminator emits light that does notintersect each other. In other words, a plurality of first light raysrepresenting the first light emitted from the first illuminator do notintersect each other. In addition, a plurality of second light raysrepresenting the second light emitted from the second illuminator do notintersect each other. Accordingly, when light is emitted from either thefirst illuminator or the second illuminator, the light from the firstilluminator or the second illuminator reaches one sensor pixel includedin the image sensor 102 from a single direction. In other words, thelight emitted from each illuminators 101 does not enter one sensor pixelof the image sensor 102 from two or more directions. The image sensor102 has a plurality of sensor pixels arranged along the light receivingsurface thereof.

Such illumination light can be realized by diffused light from theilluminators 101 each having a point-shaped light emitting unit, and canalso be realized by light from the illuminators 101 each of which emitsparallel light. For example, the illuminators 101 each having apoint-shaped light emitting unit may be substituted by a pseudo pointlight source. An example of a plurality of the pseudo point light sourceis provided by putting a light-shielding plate having a plurality ofpinholes at the vicinity of one illuminating device. Light emitted fromthe illumination device passes through the pinhole which is open andreaches the image sensor 102. The light emitted from the pinhole mimicslight emitted from the point light source. The position of the pseudopoint light source, namely, the illumination position, can be changed bychanging the pinhole to be opened. The size of each of the pinholes islimited by the pitch of the sensor pixels of the image sensor 102, thedistance between the image sensor 102 and the pinholes, and the distancefrom the image sensor 102 at which the in-focus image is generated.

The plurality of the illuminators 101 are arranged above the lightreceiving surface of the image sensor 102 and emit light downwards. Theplurality of the illuminators 101 are arranged side by side along theplane, and sequentially emit light. The plurality of the illuminators101 are arranged at different positions when viewed from the imagesensor 102, and emit light in such a way that the rays of the lighttravel from different directions to the object on the image sensor 102.For example, the plurality of the illuminators 101 may be configured asshown in FIG. 3 . FIG. 3 is a side view schematically showing oneexample of a relationship between the plurality of the illuminators 101and the image sensor 102 in the imaging device 100 according to thefirst embodiment. In this case, the plurality of the illuminators 101are arranged at different positions, for example, in a grid arrangement,on a single plane 101H which is parallel to a surface which is the lightreceiving surface of the image sensor 102. Such a plurality of theilluminators 101 emit light in such a way that the rays of the lighttravel from different directions to the object on the image sensor 102.For example, light emitted from a first illuminator 101 a and a secondilluminator 101 b included in the plurality of the illuminators 101 isincident on the object on the image sensor 102 from differentdirections. In addition, the light emitted from each of the firstilluminator 101 a and the second illuminator 101 b reach one sensorpixel of the image sensor 102 from the single direction.

As described above, the light emitted from the plurality of theilluminators 101 arranged at different positions with respect to thelight receiving surface of the image sensor 102 is incident on the lightreceiving surface at different incident angles. Furthermore, theincident direction of light with respect to the same sensor pixel of theimage sensor 102 differs for each illuminator 101. Here, theilluminators 101 are examples of a light source.

In the present embodiment, the plurality of the illuminators 101 are theplurality of the point light sources arranged on the plane 101H.However, as shown in Patent Literature 4, the plurality of theilluminators 101 may be composed of a plurality of light sources whichare arranged on a spherical surface and emit parallel light.

The image sensor 102 has a plurality of sensor pixels. Each sensor pixelof the image sensor 102 is arranged on the light receiving surface, andacquires intensity of light emitted from the plurality of theilluminators 101. The image sensor 102 acquires a captured image basedon the intensity of the light acquired by each sensor pixel. Note that“the image sensor 102 captures an image (also referred to as“photographs”)” means that the image sensor 102 detects and records theintensity of the light incident on each sensor pixel. When the spheroidis put as an object on the light receiving surface of the image sensor102, the image sensor 102 acquires the intensity of the light which haspassed through the spheroid. The image sensor 102 stores information onthe acquired captured image in the storage unit 110. An example of theimage sensor 102 is a complementary metal-oxide semiconductor (i.e.,CMOS) image sensor or a charge coupled device (i.e., CCD) image sensor.

The imaging control unit 103 controls emitting of light by the pluralityof the illuminators 101 and capturing images by the image sensor 102.Specifically, the imaging control unit 103 controls the order in whichthe plurality of the illuminators 101 emit light, the time intervals atwhich the plurality of the illuminators 101 emit light, and the like.The imaging control unit 103 associates information on the capturedimage such as an ID (Identification), a capturing time, and theilluminator which is included in the illuminators 101 and emits lightwith the captured image data captured by the image sensor 102 and storesthe information in the storage unit 110.

The imaging control unit 103 may be configured with a computer system(not shown) including a processor such as a central processing unit(i.e., CPU) or a digital signal processor (i.e., DSP), and a memory suchas a random access memory (i.e., RAM) and a read-only memory (i.e.,ROM). A part or the entire of the functions of the constituent elementsof the imaging control unit 103 may be achieved by the CPU or the DSPexecuting the program recorded in the ROM using the RAM as a temporarymemory. In addition, the part or the entire of the functions of theconstituent elements of the imaging control unit 103 may be achieved bya dedicated hardware circuit such as an electronic circuit or anintegrated circuit. The part or the entire of the functions of theconstituent elements of the imaging control unit 103 may be configuredby a combination of the above software function and hardware circuit.The program may be provided as an application through communication viaa communication network such as the Internet, communication according toa mobile communication standard, other wireless network, wired network,or broadcast. Here, the imaging control unit 103 is one example ofcontrol circuitry.

Furthermore, constituent elements other than the imaging device 100 willbe described. The storage unit 110 is realized by a storage device suchas a semiconductor memory such as a ROM, a RAM, or a flash memory, ahard disk drive, or a solid state drive (i.e., SSD), for example. Thestorage unit 110 stores the plurality of the captured images acquired bythe imaging device 100. The storage unit 110 stores the images capturedby the image sensor 102 together with position information on theilluminator which is included in the illuminators 101 and has been usedfor the capture of the images.

For example, FIG. 4 shows one example of contents stored in the storageunit 110 as described above. For each captured image file acquired bythe imaging device 100, the position information on the illuminatorwhich is included in the illuminators 101 and has been used foracquiring the captured image file, namely, the illumination position isstored. In the example of FIG. 4 , the illumination position indicatesrelative positions of the illuminators 101 with respect to the imagesensor 102. Hereinafter, the position information of the illuminators101 is also referred to as “illumination position information”, and theposition of the illuminators 101 is also referred to as “illuminationposition”. The illumination position information is stored together withor associated with the file ID of the captured image file, and iscombined with the captured image file via the file ID. Note that theillumination position information may be recorded in a part of thecaptured image file (for example, as header information).

The image processing unit 120 is realized by the control circuitry. Asillustrated in FIG. 1 , the image processing unit 120 comprises theregion extraction unit 121, the internal image generation unit 122, andthe discrimination unit 123. The image processing unit 120 may be formedby a single device or system, may be formed by a plurality of devices orsystems, and may be incorporated in a device or system other than othercomponents of the image processing unit 120.

The region extraction unit 121 extracts a region where an image of thespheroid that is an object is present from at least one captured imageincluded in a plurality of captured images which have been captured bythe imaging device 100 and stored in the storage unit 110. In otherwords, the region extraction unit 121 extracts a region of the spheroid.In the present embodiment, the region extraction unit 121 extracts theregion only from one captured image; however, is not limited to this. Animage from which the region is extracted is referred to as a “referencecaptured image”.

Specifically, in the present embodiment, from the captured imagescorresponding to the positions of the illuminators 101, the regionextraction unit 121 determines, as the reference captured image, animage captured when the illuminator which is included in theilluminators 101 and has been located immediately above the centerposition of the light receiving surface of the image sensor 102 emitslight. Furthermore, the region extraction unit 121 extracts the regionof the spheroid in the reference captured image. An extraction method ofthe region is based on, for example, a known image recognitionprocessing. The region extraction unit 121 determines, as a targetregion, a region extracted based on the result of the image recognitionfor one reference captured image. The recognition processing of theregion of the spheroid is performed based on features such as apredetermined color and outline, for example. If a plurality of regionsare extracted by recognition processing of the region of the spheroid,the region extraction unit 121 determines all of the plurality of theextracted regions as the target regions. The region extraction unit 121stores the determined target region in the storage unit 110 inassociation with the reference captured image from which the targetregion has been extracted. The reference captured image is not limitedto an image captured at the time when the illumination of theilluminators 101 located immediately above the center position of thelight receiving surface of the image sensor 102 emits light, and thereference captured image may be any captured image of the illuminators101. For example, the reference captured image may be an image capturedon the light receiving surface of the image sensor 102 at the time whenthe illumination of the illuminators 101 located immediately above aregion where the spheroid density is high.

For example, FIG. 5 illustrates one example of the contents stored inthe storage unit 110 as described above. The region extraction unit 121assigns, for example, a region ID to each of one or more spheroidregions determined as the target region. Furthermore, the regionextraction unit 121 calculates coordinates on the reference capturedimage, for example, pixel coordinates, for each region corresponding tothe region ID. The pixel coordinates are a coordinate system based onpixels in an image. As illustrated in FIG. 5 , the region extractionunit 121 associates the coordinates of each region with the region IDcorresponding to each region, and stores these in the storage unit 110.Note that any method may be used to set the coordinates of the spheroidregion. For example, the region extraction unit 121 may form arectangular frame or the like circumscribing the spheroid region on thereference captured image, and may set the coordinates of one or morepoints on the frame as the coordinates of the region. In this case, theregion extraction unit 121 may also store information on the size of theframe such as the side length in the storage unit 110 in associationwith the region ID. Alternatively, the region extraction unit 121 mayuse the coordinates of the center of gravity of the spheroid region onthe reference captured image as the coordinates of the region. In theexample of FIG. 5 , the coordinates of the spheroid region are thecoordinates of the two vertices at the diagonal position of therectangular frame.

The internal image generation unit 122 generates an internal image ofthe one or more spheroids. The internal image generation unit 122performs refocusing processing in accordance with position informationon a predetermined virtual focal plane using the plurality of thecaptured images and the illumination position information stored in thestorage unit 110, to generate an in-focus image of the spheroid at thefocal plane. The internal image generation unit 122 generates anin-focus image for each virtual focal plane. The internal imagegeneration unit 122 stores the generated in-focus image in the storageunit 110. The processing for generating the in-focus image is referredto as “refocusing processing”, and details of the refocusing processingwill be described later.

Further, FIG. 6 shows a detailed configuration of the internal imagegeneration unit 122. FIG. 6 is a block diagram illustrating one exampleof a functional configuration of the internal image generation unit 122according to the first embodiment. As shown in FIG. 6 , the internalimage generation unit 122 comprises a refocusing unit 1221, a focalplane table 1222, and an image generation unit 1223.

The focal plane table 1222 stores a position of the predeterminedvirtual focal plane. The focal plane table 1222 may have any of theconfigurations described above regarding the storage unit 110. Thevirtual focal plane is a focal plane located between the plurality ofthe illuminators 101 and the image sensor 102. In this embodiment, thevirtual focal plane is a plane parallel to the light receiving surfaceof the image sensor 102; however, may be a surface in a directionintersecting with the light receiving surface. For example, FIG. 7 showsone example of the contents stored in the focal plane table 1222. An IDis assigned to each focal plane for a plurality of the predeterminedvirtual focal planes. The distance between each focal plane and thesurface of the image sensor, that is, the light receiving surface, isstored in the focal plane table 1222 together with the ID of the focalplane. In the example of FIG. 7 , all of the virtual focal planes areplanes parallel to the surface of the image sensor 102. For example, inthe example of FIG. 7, 200 virtual focal planes are set at intervals of1 μm so as to cover the entire of the spheroid. As described above, thefocal planes may be equally or unequally spaced.

The refocusing unit 1221 generates focused pixels each forming thein-focus image on the virtual focal plane for all of one or morespheroid regions extracted by the region extraction unit 121. In thepresent embodiment, the pixels forming the in-focus image are referredto as “focused pixels”. The refocusing unit 1221 can generate thefocused pixels of the in-focus image on the focal plane from theplurality of the captured images, the position information on theplurality of the illuminators 101, and the position information on thevirtual focal plane. Specifically, the refocusing unit 1221 identifiesthe pixels in the captured image on which the focused pixels of thein-focus image are projected based on the plurality of the capturedimages and the position information on the illuminator which is includedin the illuminators 101 and illuminated when each of the captured imageswas captured. And then, the refocusing unit 1221 calculates the pixelvalue of the focused pixel using the pixel value of the identifiedpixel. The refocusing unit 1221 calculates a pixel value for eachfocused pixel. This makes it possible to generate the in-focus image.Examples of pixel values are light intensity and luminance value.

In the present embodiment, the refocusing unit 1221 does not generatefocused pixels at all pixel positions on the virtual focal plane. Therefocusing unit 1221 generates only focused pixels in all the spheroidregions. Specifically, the refocusing unit 1221 calculates the pixelcoordinates of the regions of all the spheroids on the in-focus imageson the virtual focused plane from the pixel coordinates of the regionsof all the spheroids extracted by the region extraction unit 121 asillustrated in FIG. 5 . Furthermore, the refocusing unit 1221 generatesonly focused pixels included in each region of the spheroid on thein-focus image. In the present embodiment, the pixel coordinate systemof the region of the spheroid extracted by the region extraction unit121 and the pixel coordinate system of the in-focus image of the virtualfocal plane are the same as each other. Therefore, the generationprocessing of the focused pixel is simplified.

When the focused pixel on the virtual focal plane is generated, therefocusing unit 1221 calculates a position on the image sensor 102,which light emitted by each of the illuminators 101 reaches through afocal point, from the focal point at the position of the focused pixeland the illumination position information corresponding to each capturedimage. The focal point is a point on the virtual focal plane. Further,on the captured image corresponding to each of the illuminators 101,namely, each of the illumination positions, the refocusing unit 1221extracts a pixel value at the pixel position corresponding to theposition, based on the position which the light of the illuminators 101reaches on the image sensor 102. This pixel value is a pixel valueindicating the in-focus image. Then, the refocusing unit 1221 adds allthe pixel values indicating the in-focus image extracted in the capturedimages corresponding to each illumination position. Thereby, a pixelvalue in which the luminance of all of the light which are incident fromdifferent directions and pass through the focal point has beenintegrated is provided. The pixel value is set as the pixel value of thefocused pixel. In this way, the refocusing unit 1221 generates focusedpixel information on the focal point, and performs the above-describedprocessing on each focused pixel on the in-focus image. This makes itpossible to generate the in-focus image of the region of the spheroid.The above method is the same as the refocusing technique described inPatent Literature 4. Since the technique of Patent Literature 4 isknown, the detailed description is omitted.

The image generation unit 1223 generates each of in-focus images of allregions of the spheroids on the focal planes based on the focused pixelinformation of the focal point generated by the refocusing unit 1221.The image generation unit 1223 stores the generated in-focus images inthe storage unit 110 in association with the focal plane positioninformation and the ID of the region of the spheroid corresponding tothe in-focus image.

For example, FIG. 8 illustrates one example of the contents stored inthe storage unit 110 as described above. The internal image generationunit 122 sets an ID for each in-focus image in the region of thespheroid. This ID is also associated with the focal plane including thein-focus image, and is referred to as “focal plane image ID”. For eachregion of the spheroid, the internal image generation unit 122 stores afile of the in-focus image of the region of the spheroid at each of theplurality of predetermined focal planes and the focal plane image IDcorresponding to the file in the storage unit 110 in association withthe position information on the focal plane including the in-focusimage, and in association with the ID and the coordinate of the regionof the spheroid. In the example of FIG. 8 , the focal plane positioninformation indicates the distance from the surface of the image sensor102. The focal plane position information is combined with the file ofthe in-focus image via the focal plane image ID. The focal planeposition information may be recorded in a part of the file of thein-focus image (for example, header information).

The discrimination unit 123 discriminates the outer shape of thespheroid that is the object from the cavity part in the inside of thespheroid on the basis of the pixel value for each in-focus image storedin the storage unit 110.

For example, FIGS. 9A to 9D schematically show examples of the pluralityof the in-focus images at different focal planes Fa to Fd for the samespheroid region. FIG. 9A to 9D are diagrams showing examples ofprocessed images of the region of the spheroid. The in-focus images areupper images in FIGS. 9A to 9D. The distance between the focal planes Fato Fd and the surface of the image sensor 102 are increased from FIG. 9Ato FIG. 9D. In other words, the focal plane Fa in FIG. 9A is closest tothe surface of the image sensor 102. In the in-focus image of FIG. 9A,the entire image is blurred; however, the outer shape of the spheroidcan be extracted by extracting a part darker than the periphery, namely,a part having a small pixel value. In the in-focus image of FIG. 9B, thepart darker than the periphery is extracted as the outer shape of thespheroid, and the part darker than other parts in the outer shape of thespheroid, namely, a part having a smaller pixel value can be extractedas the cavity part. In the in-focus images of FIGS. 9C and 9D, similarlyto the case of FIG. 9B, the outer shape of the spheroid and the cavitypart in the inside of the spheroid can be extracted.

The discrimination unit 123 calculates, in each in-focus image, thenumber of first pixels that are determined to be present in the insideof the outer shape of each spheroid and the number of second pixels thatare determined to be present in the cavity part in the spheroid. Thefirst pixel can include the second pixel. In one in-focus image of thespheroid, the number of the first pixels may correspond to the area inthe inside of the outer shape of the spheroid, and the number of thesecond pixels may correspond to the area of the cavity part of thespheroid. The sum of the numbers of the first pixels of all the in-focusimages corresponding to one spheroid corresponds to the volume in theinside of the outer shape of the spheroid, and the sum of the numbers ofthe second pixels of all the in-focus images corresponding to onespheroid may correspond to the volume of the cavity part of thespheroid. The discrimination unit 123 stores the numbers of the firstpixel and the second pixel of each in-focus image in the storage unit110 in association with the focal plane image ID of the in-focus image.Here, the number of the first pixels is one example of the first area,and the number of the second pixels is one example of the second area.

For example, FIG. 10 illustrates one example of the contents stored inthe storage unit 110 as described above. For each in-focus imagecorresponding to the plurality of the focal planes in the region of eachspheroid, the discrimination unit 123 stores the numbers of the firstpixels and the second pixels in association with the ID of the region ofthe spheroid, the focal plane image ID of the in-focus image, and theposition information on the focal plane of the in-focus image in thestorage unit 110. Thereby, for example, on the three-dimensionalcoordinates defined by the x-axis and y-axis, which are included in thesurface of the image sensor 102, and the z-axis, which is perpendicularto the surface, it is possible to calculate the distribution of thefirst pixels in the inside of the spheroid and the distribution of thesecond pixels of the cavity part in the spheroid.

The calculation unit 130 calculates a first total number which is thesum of the number of the first pixels at all the focal planes of all thespheroids and a second total number which is the sum of the number ofthe second pixels at all the focal planes of all the spheroids from thenumbers of the first pixels and the second pixels at each focal plane ofeach spheroid discriminated by the discrimination unit 123. Furthermore,the calculation unit 130 calculates a first ratio that is a ratiobetween the first total number and the second total number. The firstratio is indicated by the second total number/the first total number.Here, the first total number is one example of a first volume and afirst total volume, and the second total number is one example of asecond volume and a second total volume.

The state determination unit 140 compares the first ratio calculated bythe calculation unit 130 to a predetermined determination referencevalue. The state determination unit 140 determines that the culturestate of the spheroid is good, if the first ratio is lower than thedetermination reference value. On the other hand, the statedetermination unit 140 determines that the culture state of the spheroidis bad, if the first ratio is equal to or greater than the determinationreference value. The determination reference value may be set to variousvalues, depending on the type and the amount of cells forming thespheroid, the time point of the culture of the spheroid, the requiredquality of the culture state of the spheroid, and the use of thespheroid. Such a determination reference value may be determinedaccording to the above conditions by a designer, a manufacturer, or auser of the culture state determination device 10. The determineddetermination reference value may be input via an input device (notshown) and stored in the storage unit 110. In the present embodiment, aswill be described later, the spheroid is a morula of a sea urchin. Inthis case, an example of the determination reference value is 0.3.

The number of the first pixels in the inside of the spheroiddiscriminated by the discrimination unit 123 indicates the amount of thecells in the inside of the spheroid, namely, the volume of the cells.The cavity part of the spheroid is a missing part of the cells in thespheroid due to necrosis and the number of the second pixels in thecavity part indicates the amount of cells missing due to the necrosis,namely, the volume of the cells in a pseudo way. For one spheroid, asecond ratio, which is the ratio between the sum of the number of thefirst pixels at all focal planes and the sum of the number of the secondpixels at all focal planes, indicates a ratio of the amount of the cellsmissing due to the necrosis to the total amount of the cells in the onespheroid. Further, the first ratio of the first total number and thesecond total number indicates the ratio of the amount of cells missingdue to the necrosis to the total amount of the cells in the entire ofthe plurality of the photographed spheroids.

Note that the calculation unit 130 calculates the ratio between thenumber of the first pixels and the number of the second pixels; however,may calculate a difference provided by subtracting the number of thesecond pixels from the number of the first pixels. In this case, thestate determination unit 140 holds a reference number of pixels as thepredetermined determination reference. The state determination unit 140determines that the culture state is bad if the difference calculated bythe calculation unit 130 is equal to or less than the reference numberof the pixels. On the other hand, the state determination unit 140determines that the culture state is good, if the difference exceeds thereference number of the pixels. The difference provided by subtractingthe number of the second pixels from the number of the first pixelsshows a pseudo amount of normal cells that are not necrotized containedin the spheroid. Also in this case, the calculation unit 130 maycalculate a difference provided by subtracting the second total numberfrom the first total number. Similarly to the case of the differencebetween the number of the first pixel and the number of the secondpixel, the state determination unit 140 determines whether the culturestate is good or bad for the entire spheroid based on the differencebetween the first total number and the second total number. In such acase, the state of the spheroids in the entire culture vessel may bedetermined, and whether or not the spheroids in the culture vessel canbe used may be determined. The determination based on the difference asdescribed above is effective, if the number of the spheroids in theculture vessel is known. In addition, the reference of the differencemay be based on the determination reference of the first ratio.

The display unit 150 shows the result determined by the statedetermination unit 140. Examples of the display unit 150 are a displayand a speaker. Examples of the display are a liquid crystal panel and anorganic or inorganic electroluminescence (i.e., EL). If the display unit150 is a display, the result can be displayed by characters, symbols,images, and the like. In addition, if the display unit 150 is a speaker,the result may be indicated by a sound and an acoustic signal. Thedisplay unit 150 may include one or both of a display and a speaker. Thedisplay unit 150 may be other display output means. For example, thedisplay unit 150 may have a configuration that projects onto a wallsurface, a glass surface, a space, or the like.

1-2. Operation of Culture State Determination Device According to FirstEmbodiment

The operation of the culture state determination device 10 according tothe first embodiment will be described with reference to FIG. 11 . FIG.11 is a flowchart showing one example of the operation of the culturestate determination device 10 according to the first embodiment.

First, in the step S1100, the imaging control unit 103 of the imagingdevice 100 uses the plurality of the illuminators 101 sequentially toilluminate the plurality of the spheroids that are objects on the imagesensor 102, and causes the image sensor 102 to capture a plurality ofimages of the plurality of the spheroids. For example, as shown in FIG.12 , the plurality of the spheroids are present in a culture solution 2in a culture vessel 1 such as a well placed on the image sensor 102.FIG. 12 is a schematic perspective view showing one example of theimaging state of spheroids. The imaging control unit 103 causes theimage sensor 102 to record the intensity of light which has reached thelight receiving surface every time when each of the plurality of theilluminators 101 illuminates the spheroid, to acquire a plurality ofcaptured images on which the entire of the plurality of the spheroids inthe culture vessel 1 have been photographed. The imaging control unit103 stores the captured images in the storage unit 110 in associationwith the position information on the illuminators 101 illuminated whenthe captured image is captured. In the present embodiment, the positionsof the plurality of the illuminators 101 are fixed with respect to theimage sensor 102, and the position information on each of the pluralityof the illuminators 101 is determined in advance and stored in thestorage unit 110. Details of the imaging processing will be describedlater.

Next, in the step S1200, the region extraction unit 121 of the imageprocessing unit 120 extracts a region where a spheroid image has beencaptured, namely, a region of the spheroid, from the captured imageacquired in the step S1100. Specifically, the region extraction unit 121determines one captured image as the reference captured image from amongthe plurality of the captured images acquired in the step S1100 andstored in the storage unit 110, and acquires the reference capturedimage and the illumination position information corresponding to thereference captured image from the storage unit 110. The illuminationposition information is position information on the illuminators 101illuminated when the reference captured image is captured. The referencecaptured image is, for example, a captured image at the time ofillumination of the illuminators 101 located immediately above thecenter point of the light receiving surface of the image sensor 102. Theregion extraction unit 121 extracts one or more spheroid regions basedon the pixel value of each pixel in the reference captured image.

An example of the extraction method is to binarize the referencecaptured image based on a first threshold value set between the maximumvalue and the minimum value of the pixel values of the referencecaptured image, and then to divide into a region where the light emittedfrom the illuminators 101 has reached the light receiving surface of theimage sensor 102 directly and a region where the light has passedthrough the spheroid and has reached the light receiving surface of theimage sensor 102.

The first threshold value is a threshold value for distinguishing theregion where the spheroid has been photographed from the region wherethe background of the spheroid has been photographed. The firstthreshold value may be determined to have various values, depending onconditions such as the type and the amount of the cells forming thespheroid, the time of the culture of the spheroid, and the environmentduring the capturing of the images. Such a first threshold value may bedetermined according to the above conditions by the designer, themanufacturer, the user, or the like of the culture state determinationdevice 10, and the determined first threshold value may be input via aninput device (not shown) and stored in the storage unit 110. Forexample, the first threshold value is approximately 50% to 70% of thepixel value between the minimum value and the maximum value of the pixelvalue of the reference captured image. For example, when (pixel valuemaximum value)−(pixel value minimum value)=α for all the pixels includedin the reference captured image, the first threshold value is may bedetermined within a range of 0.5α+(the minimum value of the pixelvalue)≤(the first threshold value)≤0.7α+(the minimum value of the pixelvalue). In addition, in a histogram of the luminance values of thepixels, which is one example showing the distribution of the pixelvalues in the reference captured image, a pixel having a luminance valuewith a sharp increase in the number of the pixels may indicate thebackground of the spheroid. The first threshold value may be determinedto be a value equal to or less than such luminance values. The firstthreshold value may be a value, for example, provided by multiplying themaximum value of the pixel value of the reference captured image by apredetermined ratio. Such a ratio is a value greater than 0 and lessthan 1, and an example of the ratio is 0.6; however, is not limitedthereto.

The region where the image is brighter, namely, the pixel value is equalto or greater than the first threshold value is a region where the lightemitted from the illuminators 101 has reached the light receivingsurface of the image sensor 102 directly. The region where the image isdarker, namely, the pixel value is smaller than the first thresholdvalue is a region where the light has passed through the spheroid andhas reached the light receiving surface of the image sensor 102. In thebinarized reference captured image, a region where pixels each having apixel value smaller than the first threshold value continue isdetermined as a region where the spheroid has been photographed.

The region extraction unit 121 extracts a region where the pixels eachhaving a pixel value smaller than the first threshold value continues inthe reference captured image, and determines a minimum rectangularregion including the region, for example, a rectangular regioncircumscribing the region as the region where the image processing willbe performed. Furthermore, the region extraction unit 121 sets an ID forthe determined rectangular region, and calculates pixel coordinates onthe reference captured image of the rectangular region. The regionextraction unit 121 stores the pixel coordinates of the rectangularregion and the ID of the rectangular region in the storage unit 110 inassociation with each other. Note that the region extraction unit 121may calculate the pixel coordinates of at least one vertex of therectangular region as the pixel coordinates of the rectangular region.Furthermore, the region extraction unit 121 may store the length of theside of the rectangular region in the storage unit 110 as the dimensionsof the rectangular region together with the pixel coordinates of therectangular region. Since one or more spheroids are present in theculture vessel 1, the region of each of the spheroids and therectangular region thereof are extracted from the reference capturedimage. In other words, the region of the one or more spheroids and therectangular regions thereof are extracted.

In the step S1200, the region extraction unit 121 determines a regionwhere pixels each having a pixel value smaller than the first thresholdvalue in the binarized reference captured image as a region where thespheroid has been photographed; however, the region where the spheroidhas been photographed may be determined by another method. For example,the region extraction unit 121 may perform edge extraction using adifference in the pixel values between the pixels in the referencecaptured image, and determine a region surrounded by the edges as aregion where the spheroid has been photographed. Alternatively, forexample, the region extraction unit 121 may extract a region wherepixels each having similar pixel values continue by clustering with thepixel values of the pixels in the reference captured image, anddetermine the extracted region as a region where the spheroid has beenphotographed.

Next, in the step S1300, the internal image generation unit 122 of theimage processing unit 120 generates the in-focus images at a pluralityof the predetermined focal planes, using the plurality of the capturedimages acquired in the step S1100, for all of the one or more regions ofthe spheroid determined in the step S1200, namely, for all rectangularregions. In other words, an in-focus image of each rectangular region oneach focal plane is generated. Since such an internal image generationunit 122 does not generate an in-focus image in a region other than therectangular regions, the processing speed for generating the in-focusimages can be improved. In the present embodiment, all of the pluralityof focal planes are flat surfaces, and each focal plane is parallel tothe other focal planes. Further, the plurality of the focal planes areparallel to the light receiving surface of the image sensor 102;however, are not limited thereto. The positions of the plurality of thefocal planes are defined using, for example, the distance from the lightreceiving surface of the image sensor 102 and stored in the storage unit110 in advance. The plurality of the focused pixels included in thein-focus image on the focal plane correspond one-to-one to a pluralityof points on the focal plane. A method for generating an in-focus imagewill be described later.

Next, in the step S1400, the discrimination unit 123 of the imageprocessing unit 120 extracts the outer shape of the spheroid for each ofall the in-focus images generated in the step S1300, based on the pixelvalue of the in-focus image, and extracts the cavity part in the insideof the outer shape of the spheroid. The discrimination unit 123discriminates a region of a pixel that is distinguished from otherpixels as the cavity part in the inside of the outer shape of thespheroid. The cavity part can be discriminated by, for example, thedistribution of the pixel values in the in-focus image.

For example, each of FIGS. 9A to 9D shows an example of a series ofprocessed images of the region of the spheroid. In each of FIGS. 9A to9D, an in-focus image of the region of the spheroid is shown in theupper part. In the middle part, a binarized image that is a binarizedimage of the in-focus image in the upper part is shown. In the binarizedimage, a region where the pixel value is equal to or greater than asecond threshold value is indicated as a white region or a non-coloredregion, and a region where the pixel value is less than the secondthreshold value is indicated as a black region.

Note that the second threshold value is a threshold value fordistinguishing a region where a cell is photographed from a region wherethe cavity part is photographed in the spheroid. In the binarized imagesof FIGS. 9A to 9D, the black regions may indicate the cavity parts, andthe white regions or the non-colored regions surrounded by the blackregions may indicate cells. The white regions or the non-colored regionsmay indicate cells, and the black regions may indicate the cavity parts.The second threshold value can be determined to have various values,depending on conditions such as the type and the amount of the cellsforming the spheroid, the time of the culture of the spheroid, and theenvironment during the capturing of the images. Such a second thresholdvalue may be determined according to the above conditions by a designer,a manufacturer, or a user of the culture state determination device 10.The determined second threshold value may be input via an input device(not shown) and be stored in the storage unit 110. For example, thesecond threshold value is a pixel value between approximately 50% to 70%of the pixel value between the minimum value and the maximum value ofthe pixel value of the in-focus image. For example, with respect to allthe pixels included in the in-focus image in the region of the spheroid,when (pixel value maximum value)−(pixel value minimum value)=β, thesecond threshold value can be determined within a range of 0.5β+(theminimum value of the pixel value)≤(the second threshold value)≤0.7β+(theminimum value of the pixel value).

In the lower part, an extracted image, which is an image provided byextracting a region of the cell in the binarized image in the middlepart, is shown. The image in the lower part schematically shows an imageprovided by separating regions where the pixels having the pixel valuescorresponding to the cells are continuous from the binarized image. Asdescribed above, FIGS. 9A to 9D are images corresponding to differentfocal planes Fa to Fd for the same spheroid region as each other,respectively. The spheroid of FIGS. 9A-9D is a sea urchin morulae. Themorula is a cell aggregation composed of a plurality of cells of almostthe same size as each other, and includes the cavity part at the center.

When discriminating the cavity part in the spheroid, the discriminationunit 123 binarizes the in-focus image as shown in the upper part ofFIGS. 9A to 9D to generate a binarized image in the middle part.Further, the discrimination unit 123 labels a region where the pixelvalue is equal to or greater than the second threshold value on thebinarized image, namely, labels the region, and determines a pluralityof regions of the cells such as the lower image. In other words, theimages in the lower parts are extracted images of the cell region.

Here, an example of a method for determining the presence or absence ofthe cavity part from the binarized image as shown in FIGS. 9A to 9D willbe described with reference to FIG. 13 . FIG. 13 is a diagramschematically illustrating a method of determining the presence orabsence of the cavity part in the binarized image of the in-focus imageof the spheroid. The discrimination unit 123 labels regions where thepixel value is equal to or greater than the second threshold value inthe binarized image of the in-focus image of the spheroid Sp, anddetermines a first region La to a seventh region Lg, which are sevenregions. Further, the discrimination unit 123 determines a centroid Gcalculated from all of the first region La to the seventh region Lg. Forexample, the centroid G is a centroid of the seven centroids of thefirst region La to the seventh region Lg. Then, the discrimination unit123 forms a circle Cb. The circle Cb has a radius b and has the centroidG as the center thereof. The radius b is, for example, an average of theradii of seven approximate circles when the outer shape of each of thefirst region La to the seventh region Lg is approximated as a circle. Ifthe circle Cb includes any of the labeled regions, namely, any of thefirst region La to the seventh region Lg, the discrimination unit 123determines that the cavity part is absent. On the other hand, if thecircle Cb includes none of the first region La to the seventh region Lg,the unlabeled region including the center of gravity G is defined as thecavity part. The unlabeled region may be any one of a region whichincludes the centroid G and is a region other than the first region Lato the seventh region Lg, the circle Cb, or an ellipse which isinscribed in the first region La to the seventh region Lg and includesthe centroid G. In the present specification and claims, an “ellipse”includes a circle, an ellipse, or an oval. Also, the circle Cb includesany one of the first region La to the seventh region Lg means that thecircle Cb and any one of the first region La to the seventh region Lgform an overlapping region.

In the present embodiment, the discrimination unit 123 determines alabeled region from the binarized image of the in-focus image of thespheroid, and determines a non-labeled region as the cavity part. Thecavity part may be determined by another method. For example, thediscrimination unit 123 may determine the cavity part based on a changein the pixel values on a line that crosses the spheroid in the in-focusimage. For example, FIGS. 14A to 14D schematically show examples of therelationship between the in-focus images of the same spheroid regions aseach other on the focal planes Fa to Fd and the pixel values on theregions, similarly to the case in FIGS. 9A to 9D. Specifically, thegraphs in the lower parts of FIGS. 14A to 14D schematically show thepixel values of the pixels on a straight line L which passes through thecenter of the spheroid in the in-focus image in the upper part.

When determining the presence or absence of the cavity part, thediscrimination unit 123 determines, for example, the straight line Lwhich passes through the center of the spheroid for each in-focus imageas shown in the upper parts of FIGS. 14A to 14D. A distribution of thepixel values of the pixels along the straight line L is provided by thediscrimination unit 123 as shown in the lower parts. The position of thestraight line L is preferably determined so as to pass through a regionwhere the cavity part is likely to be formed. Since a morula includesthe cavity part at the center thereof, the straight line L passesthrough the center of the spheroid in the present embodiment. Thus, theposition of the straight line L can be determined depending on a targetcell aggregation of the spheroid.

In the graph of the pixel values on the straight line L as shown in thelower parts of FIGS. 14A to 14D, the discrimination unit 123 calculatesthe interval between the tops of the peaks and the valleys of the graph,namely, the interval between the peaks and valleys of the pixel values.In FIG. 14A to 14D, the positions of the tops of the valleys of thegraph are indicated by broken lines extending from the graph to thein-focus image, and the positions of the tops of the peaks of the graphare indicated by dotted lines extending from the graph to the in-focusimage. The broken line indicates the position in the in-focus imagecorresponding to the peak of the valley of the graph. The dotted lineindicates the position in the in-focus image corresponding to the top ofthe peak of the graph. For example, the discrimination unit 123calculates eight intervals in the example of FIG. 14A, calculates eightintervals in the example of FIG. 14B, calculates twelve intervals in theexample of FIG. 14C, and calculates twelve intervals in the example ofFIG. 14D. Furthermore, the discrimination unit 123 calculates thevariance of the intervals between the peaks and valleys of the pixelvalues for each in-focus image.

The discrimination unit 123 determines that the cavity part is absent ifthe variance of the intervals between the peak and valley of the pixelvalues is less than a predetermined third threshold value, anddetermines that the cavity part is present if the variance of theintervals between the peaks and valleys is equal to or greater than thethird threshold value. Furthermore, if the cavity part is present, thediscrimination unit 123 determines the region which is along thestraight line L and has the largest interval between the peaks andvalleys as the region of the cavity part. For example, thediscrimination unit 123 determines that the cavity part is present inFIGS. 14B and 14C. The discrimination unit 123 determines the region Abas the cavity part in FIG. 14B, and determines the region Ac as thecavity part in FIG. 14C. In addition, the discrimination unit 123further determines a plurality of straight lines that pass through thecenter of the spheroid and are different from the straight line L. Thediscrimination unit 123 determines the presence or absence of the cavitypart on the basis of the distribution of the pixel values along eachline, namely, on the basis of the variance of the intervals between thepeaks and valleys of the pixel values. The discrimination unit 123determines the region of the cavity part along the straight lines. Theplurality of the straight lines are straight lines each intersectingwith the straight line L, and are straight lines provided by rotatingthe straight line L at the center of the spheroid. The discriminationunit 123 calculates a two-dimensional region of the cavity part alongthe in-focus image from a one-dimensional region of the cavity partalong each of the plurality of the straight lines including the straightline L. For example, the discrimination unit 123 may calculate thetwo-dimensional region of the cavity part by integrating theone-dimensional region of the cavity part.

The third threshold value is a threshold value for determining thepresence of the cavity part in the spheroid. The third threshold valuecan be determined as various values depending on conditions such as thetype and the amount of the cells forming the spheroid and the time pointof the culture of the spheroid. Such a third threshold value may bedetermined depending on the above conditions by a designer, amanufacturer, or a user of the culture state determination device 10,and the determined third threshold value may be input via an inputdevice (not shown) and may be stored in the storage unit 110. Forexample, the variance of the intervals between the peaks and valleys ofthe pixel value in a case where the cavity part has a size which is notless than approximately twice as large as the size of the cell is notless than four times as much as the variance of the intervals betweenthe peaks and valleys of the pixel value in a case where the cavity parthaving a size which is equal to or larger than the size of the cells isabsent. If the region that is not less than approximately two times aslarge as the cell is deemed to be a cavity part, an example of the thirdthreshold value is not less than four times as much as the variance ofthe intervals between the peaks and valleys of the pixel value if thecavity part having a size equal to or larger than the size of the cellis absent. However, the third threshold value is not limited to such avalue, and may be variously determined based on the relationship betweenthe size of the region which is deemed to be the cavity part and thesize of the cells.

Next, in the step S1500, the calculation unit 130 determines the numberof the first pixel that is a pixel in the region surrounded by the outershape of the spheroid and the number of the second pixel which is apixel in the region of the cavity part, for the regions of the outershape and the cavity part of the spheroid in all the focal planes foreach of the regions of the spheroids discriminated in the step S1400.The number of the first pixel is the number of all the pixels includedin the outer shape of the spheroid, and the number of the second pixelis the number of all the pixels included in the cavity part.Furthermore, the calculation unit 130 calculates the first total numberthat is the sum of the number of the first pixels in the regionsurrounded by the outer shape of the spheroid at all the focal planes ofall the regions of the spheroid. Further, the calculation unit 130calculates the second total number that is the sum of the number of thesecond pixels in the region of the cavity part of the spheroid at allthe focal planes of all the regions of the spheroid. The calculationunit 130 calculates a first ratio of the second total number to thefirst total number. The first ratio is indicated by the second totalnumber/the first total number. The number of the pixels indicate an areain which the area of one pixel is one unit. The sum of the areasindicated by the number of the pixels on a plurality of parallel focalplanes indicates the volume of a three-dimensional region including thepixels in a pseudo manner. Since the spheroid is a mass in which cellsare densely aggregated, the ratio of the number of the pixels indicatesthe ratio of the amount of cells in a pseudo manner in the inside of theouter shape of the spheroid. The amount of the cells may mean the volumeof the cells or the number of the cells. Thus, the calculation unit 130calculates the pseudo cell amount of the spheroid.

Next, in the step S1600, the state determination unit 140 determines thestates of the plurality of the cultured spheroids based on the firstratio calculated in the step S1500. If a ratio of the second totalnumber that is the sum of the number of the second pixels included ineach of the cavity parts of all the spheroids to the first total numberthat is the sum of the number of the first pixels included in each ofthe outer shapes of all the spheroids is large, the culture state isallowed to be determined to be bad. Specifically, the statedetermination unit 140 determines that the culture state is good, if thefirst ratio is lower than a predetermined determination reference value.On the other hand, the state determination unit 140 determines that theculture state is bad, if the first ratio is equal to or higher than thedetermination reference value. In the present embodiment, thedetermination reference value is 0.3. If the second total number is 30%or more of the first total number, the state determination unit 140determines that the culture state is bad and determines to discard allthe plurality of the spheroids in the culture vessel 1. If the secondtotal number is less than 30% of the first total number, the statedetermination unit 140 determines that the culture state is good, anddetermines to be used for the processing after the culture. As describedabove, the state determination unit 140 determines a state in which aplurality of spheroids contain more cells as a good culture state. Inother words, a good culture state is a culture state in which there aremore cells that can be used for the processing after the culture, and anefficient processing after the culture is possible.

Next, in the step S1700, the display unit 150 shows the determinationresult of the step S1600 to the user. At this time, the display unit 150shows the output of images, characters, sounds, and the like via adisplay and/or a speaker.

1-3. Imaging Processing

Details of the operation of the imaging device 100 in the step S1100will be described with reference to FIG. 15 . FIG. 15 is a flowchartillustrating one example of the operation of the imaging device 100.

In the step S1110, the imaging control unit 103 determines whether ornot capturing of the image of the plurality of the spheroids illuminatedfrom the position of each of the illuminators 101 is completed withreference to, for example, a list of positions of the predeterminedplurality of the illuminators 101 stored in the storage unit 110 or alist of positions of the plurality of the illuminators 101 designated byan external input which is not shown (hereinafter, each of the lists isreferred to as “illumination position list”).

Here, if the capturing of the image with illumination from allillumination positions included in the illumination position list hasbeen completed (Yes in step S1110), the imaging control unit 103proceeds to the step S1200. On the other hand, if the capturing of theimages with illumination from any illumination positions in theillumination position list has not been completed (No in step S1110),the imaging control unit 103 proceeds to the step S1120.

Next, in the step S1120, the imaging control unit 103 selects anillumination position that has not yet been illuminated from among theplurality of the illumination positions included in the illuminationposition list, and outputs a control signal to the illuminator 101 atthe selected illumination position. In the illumination position list,each illumination position is indicated by, for example, a numberassigned to each illumination position. Alternatively, each illuminationposition is indicated by, for example, the coordinate value of thethree-dimensional coordinate space defined by the x-axis and y-axiswhich are included in the light receiving surface of the image sensor102, and the z-axis perpendicular to the light receiving surface. Theselection of the illumination position is performed, for example, inascending order of the list.

Next, in the step S1130, the illuminator 101 starts illumination of theplurality of the spheroids in the culture vessel 1 on the image sensor102 in accordance with the control signal output from the imagingcontrol unit 103 in the step S1120. In other words, the illuminator 101at the illumination position selected in the step S1120 startsillumination of light.

Next, in the step S1140, while the plurality of the spheroids areilluminated by the illuminator 101, the imaging control unit 103 causesthe image sensor 102 to acquire a captured image formed by the lightemitted from the illuminator 101. The captured image includes an imageformed by the light transmitted through the spheroids.

Next, in the step S1150, the imaging control unit 103 outputs a controlsignal to the illuminators 101, and stops the illumination on thespheroids. The stop of the illumination does not have to be performed inaccordance with a control signal from the imaging control unit 103. Forexample, the illuminator 101 may measure the time length from the startof the illumination and actively stop the illumination, when themeasured time length exceeds a predetermined time length. Alternatively,after the image sensor 102 finishes acquiring the captured image in thestep S1140, the image sensor 102 may output a control signal to stop theillumination to the illuminator 101.

Next, in the step S1160, the imaging control unit 103 stores thecaptured image acquired in the step S1140 and the position informationof the illuminator 101 used in the step S1130 in the storage unit 110 inassociation with each other. The imaging control unit 103 returns to thestep S1110 after the processing of the step S1160.

The imaging control unit 103 repeats the processing from the step S1110to the step S1160 to sequentially irradiate the spheroids with lightfrom the illuminators 101 at all the illumination positions included inthe illumination position list. In this way, the imaging control unit103 acquires the captured image every time when the spheroids areirradiated with light.

1-4. Refocusing Processing

Details of the operation of the refocusing unit 1221 in the step S1300will be described with reference to FIG. 16 . FIG. 16 is a flowchartshowing one example of the operation of the refocusing unit 1221according to the first embodiment.

In the step S1310 which follows the step S1200, the refocusing unit 1221acquires a list of one or more extraction regions determined using thereference captured image in the step S1200, namely, a list of theregions each including the image of the spheroid from the storage unit110. In the following description, the extraction region and the regionincluding the image of the spheroid are referred to as a “spheroidregion”. The list is, for example, a list as shown in FIG. 5 .

Next, in the step S1320, the refocusing unit 1221 refers to the list ofthe spheroid regions acquired in the step S1310, and determines whetheror not the refocusing processing for all the spheroid regions has beencompleted. The completion of the refocusing processing for all thespheroid regions means the completion of a series of the processing inthe steps S1320 to S1370. In other words, the completion of therefocusing processing for all the spheroid regions means the processingof generating the in-focus images at all the predetermined focal planesusing the plurality of the captured images is completed for eachspheroid region.

If the refocusing processing has been completed for all the spheroidregions included in the list of the spheroid regions (Yes in stepS1320), the refocusing unit 1221 proceeds to the step S1400. On theother hand, if the refocusing processing for any spheroid region in thelist of the spheroid regions has not been completed (No in step S1320),the refocusing unit 1221 proceeds to the step S1330.

Next, in the step S1330, the refocusing unit 1221 selects one spheroidregion that has not been refocused, namely, an extraction region, fromthe list of the spheroid regions acquired in the step S1310. Therefocusing processing of the spheroid region is a series of processingin the steps S1340 to S1370.

Next, in the step S1340, the refocusing unit 1221 determines whether ornot the generation of the in-focus images at all focal planes has beencompleted for the selected spheroid region with reference to the focalplane table 1222 that has stored information on the plurality of thepredetermined focal planes and the list of the spheroid regions acquiredin the step S1310.

If the generation of the in-focus images on all focal planes stored inthe focal plane table 1222 has been completed (Yes in step S1340), therefocusing unit 1221 returns to step S1320. On the other hand, if thegeneration of the in-focus images on all the focal planes stored in thefocal plane table 1222 has not been completed (No in step S1340), therefocusing unit 1221 proceeds to the step S1350.

Next, in the step S1350, the refocusing unit 1221 selects one focalplane that has not yet generated a corresponding in-focus image from thefocal planes stored in the focal plane table 1222.

Next, in the step S1360, the refocusing unit 1221 performs refocusingprocessing on the focal plane selected in the step S1350 using theplurality of the captured images acquired in the step S1100 for thespheroid region selected in the step S1330, and generates an in-focusimage of the spheroid region on the focal plane.

For example, the refocusing unit 1221 performs the refocusing processingin the same manner as in Patent Literature 4. The in-focus imageincludes a plurality of focused pixels. The plurality of the focusedpixels included in the in-focus image correspond one-to-one to aplurality of points on the focal plane. According to the same method asin Patent Literature 4, the refocusing unit 1221 calculates a pointcorresponding to the spheroid region on the focal plane, and furthercalculates the pixel coordinates of the focused pixel corresponding tothe point. Further, the refocusing unit 1221 calculates the lightarrival position on the light receiving surface of the image sensor 102when light emitted from the plurality of the different illuminationpositions passes through the position of the focused pixel and reachesthe light receiving surface of the image sensor 102. The refocusing unit1221 calculates the position of the point where the illumination lightthat has passed through the position of the focused pixel reaches theimage sensor 102 for each of the plurality of the different illuminationpositions for one focused pixel. The refocusing unit 1221 acquires pixelvalues acquired by the image sensor 102 at the position of the arrivalpoint from the plurality of the captured images. Specifically, therefocusing unit 1221 acquires the pixel value at the pixel coordinatesof the arrival point of the light from the illumination position in thecaptured image corresponding to each illumination position. Further, therefocusing unit 1221 calculates the pixel value of the focused pixel byadding the pixel values at the arrival points on the image sensor 102acquired for all the illumination positions with respect to the focusedpixel. In other words, the refocusing unit 1221 calculates the pixelvalue of the focused pixel by adding the pixel value at the pixelcoordinate of the arrival point acquired in each captured imagecorresponding to all the illumination positions. Further, the refocusingunit 1221 performs the above calculation for all focused pixels on thefocal plane on which an in-focus image is to be generated, namely, forall focused pixels corresponding to the spheroid region.

Next, in the step S1370, the image generation unit 1223 generatesin-focus image data of the spheroid region on the basis of the pixelvalue for each focused pixel on the in-focus image generated in the stepS1360, namely, generates the image data of the spheroid region in thefocal plane corresponding to the in-focus image. Further, the imagegeneration unit 1223 stores the in-focus image data of the spheroidregion in the storage unit 110 in association with the information onthe spheroid region and the position information on the focal planecorresponding to the in-focus image. The image generating unit 1223returns to the step S1340 after the step S1370 is completed.

As described above, by repeating the processing from the step S1340 tothe step S1370, in-focus images on all focal planes stored in the focalplane table 1222 are generated for the spheroid region selected in thestep S1330.

Further, by repeating the processing from the step S1320 to the stepS1370, the in-focus images on all focal planes stored in the focal planetable 1222 are generated for all the spheroid regions extracted in thestep S1200.

Here, a specific example of the calculation method of the refocusingprocessing will be described with reference to FIGS. 17-20 . In thisembodiment, the focal plane is a plane parallel to the light receivingsurface of the image sensor 102. In the following, a case where thefocal plane intersects the light receiving surface of the image sensor102 will be described. The specific calculation method is the same asone another in all cases. For example, FIG. 17 illustrates one exampleof a positional relationship among the plurality of the illuminators 101of the imaging device 100, a spheroid 1000, and the image sensor 102.FIG. 17 shows one example of a cross-sectional view of the image sensor102 and the spheroid 1000 in a plane perpendicular to the lightreceiving surface of the image sensor 102. The spheroid 1000 is locatedbetween the illuminators 101 a and 101 b and the image sensor 102 and islocated on the image sensor 102. A focal plane 1100 for generating anin-focus image passes through the spheroid 1000 and intersects the lightreceiving surface of the image sensor 102.

FIG. 18 shows one example of a plurality of points 1102 a to 1102 e onthe focal plane 1100 corresponding to a plurality of focused pixelsincluded in the in-focus image, similarly to the case of FIG. 17 . Amethod for generating a focused pixel corresponding to the point 1102 aamong the plurality of the points 1102 a to 1102 e will be described.Since the method for generating focused pixels corresponding to otherpoints is the same as that of the point 1102 a, the description thereofis omitted. FIG. 19 shows an example in which light emitted from each ofthe illuminators 101 a and 101 b passes through the point 1102 a on thefocal plane and is received by the image sensor 102.

The light emitted from the illuminators 101 a and passed through thepoint 1102 a travels on a straight line 1200 a passing through theposition of the illuminator 101 a and the point 1102 a, and reaches anintersection 1103 a between the straight line 1200 a and the lightreceiving surface of the image sensor 102. The luminance value of thelight reaching the intersection 1103 a from the illuminators 101 a isincluded in the captured image of the image sensor 102 when theilluminator 101 a is illuminated. In the captured image, a pixel at aposition corresponding to the intersection 1103 a includes an image atthe point 1102 a on the focal plane 1100, namely, a luminance value. Theposition of the intersection 1103 a can be calculated from the positionof the illuminator 101 a and the position of the point 1102 a.

The light emitted from the illuminator 101 b and passed through thepoint 1102 a travels on a straight line 1200 b passing through theposition of the illuminators 101 b and the point 1102 a, and reaches anintersection 1103 b between the straight line 1200 b and the lightreceiving surface of the image sensor 102. The luminance value of thelight reaching the intersection 1103 b from the illuminators 101 b isincluded in the captured image of the image sensor 102 when theilluminator 101 b is illuminated. In the captured image, a pixel at aposition corresponding to the intersection 1103 b includes an image atthe point 1102 a on the focal plane 1100, namely, a luminance value. Theposition of the intersection 1103 b can be calculated from the positionof the illuminator 101 b and the position of the point 1102 a.

By adding the luminance value of the image at the intersection 1103 aand the luminance value of the image at the intersection 1103 b, aplurality of images formed by light from a plurality of directions aresuperimposed on the focused pixel at the point 1102 a on the focal plane1100. A focused pixel at the point 1102 a is generated by superimposinga plurality of images formed by light transmitted from all theilluminators 101 through the point 1102 a. In this way, by using theluminance values of each of the sensor pixels in a condition where inwhich the position of the illuminators 101, the position of the focusedpixel, and the position of the sensor pixel of the image sensor 102 arealigned on a straight line, the luminance value of the focused pixel iscalculated.

If the position of the intersection in the captured image matches theposition of the pixel in the captured image, the luminance value of thepixel may indicate the luminance value of the intersection. If theposition of the intersection in the captured image is an intermediateposition between the plurality of pixels in the captured image, theluminance value of the intersection in the captured image may becalculated by performing an interpolation processing using the luminancevalues of the plurality of pixels adjacent to the position of theintersection. Specifically, for example, as shown in FIG. 20 and thefollowing formula 1, for a plurality of pixels (for example, fourpixels) adjacent to the intersection, the ratio of a reference distanceto the distance between each pixel and the intersection is multiplied bythe luminance value of each pixel, and added. In this way, the luminancevalue of the intersection in the captured image can be provided. In FIG.20 , the distances between the four pixels A to D adjacent to theintersection and the intersection are represented as a, b, c, and d,respectively. In this case, the luminance value Lt of the intersectioncan be calculated in accordance with the following formula 1.

$\begin{matrix}{{Lt} = {\frac{1}{4}\left( {\frac{La}{a} + \frac{Lb}{b} + \frac{Lc}{c} + \frac{Ld}{d}} \right) \times S}} & \left\lbrack {{Formula}\mspace{14mu} 1} \right\rbrack\end{matrix}$

where, La, Lb, Lc, and Ld represent the luminance values of the pixel A,the pixel B, the pixel C, and the pixel D, respectively. S represents areference distance. For example, S may be an average of the distancebetween the intersection and each pixel as shown in the followingformula 2.

$\begin{matrix}{S = \frac{a + b + c + d}{4}} & \left\lbrack {{Formula}\mspace{14mu} 2} \right\rbrack\end{matrix}$

1-5. Effect

As described above, the culture state determination device 10 accordingto the first embodiment performs the refocusing processing using theplurality of the captured images each having different illuminationposition from one another at the time when the images are captured togenerate the in-focus images of the plurality of the spheroids at eachof the plurality of the parallel focal planes and to discriminate theouter shape of the spheroid from the cavity part of the inside of thespheroid on each in-focus image. Furthermore, the culture statedetermination device 10 calculates the number of the first pixelsforming the inner region of the outer shape of each spheroid and thenumber of the second pixels forming the cavity part on all focal planes,and calculates the volume of the spheroid and the volume of the cavitypart each having the pixel as a unit. Thereby, the amount of the cellsforming the spheroid is allowed to be calculated in a pseudo manner. Fora culture vessel containing a plurality of spheroids, the state wherethe first ratio of the volume of the cavity part to the volume of thespheroids in the entire culture vessel is small is a state where theamount of the cells provided as a result of culture is relatively large.Such a state can be determined as the state where the culture state isgood. Since the culture state determination device 10 can determine thequality of the culture state based on not the quality of the culturestate of individual spheroid but the efficiency of the culture in theentire culture vessel, the efficiency of acquiring cells that can beused for the processing after the culturing can be improved, and theamount of the provided cells can be increased. As just described, theculture state determination device 10 simultaneously captures the imagesof the plurality of the spheroids in the same culture vessel, andevaluates the state of the inside of all the spheroids. In this way, theculture state determination device 10 determines the quality of theculture state of the entire spheroids contained in the same culturevessel, and allows usable spheroids to be selected.

In the culture state determination device 10 according to the firstembodiment, in the step S1200, the region extraction unit 121 extractsthe region in which a spheroid has been photographed from the capturedimage, and, in the step S1300, the internal image generation unit 122performs the refocusing processing for each extracted region to generatean in-focus image at each focal plane. However, the present disclosureis not limited to this. The culture state determination device 10 doesnot perform the region extraction in the step S1200, and the internalimage generation unit 122 may set the range of the light receivingsurface of the image sensor 102 to be the xy plane direction in the stepS1300, for example, and may perform the refocusing processing for allthe pixels in the three-dimensional space in which the z axis isorthogonal to the xy plane. In this case, the discrimination unit 123extracts the outer shape of the plurality of the spheroids in thethree-dimensional space, and discriminates the cavity part in the outershape of each of the spheroids. Further, the calculation unit 130calculates the number of the first pixels included in the outer shape ofeach of the plurality of the spheroids and the number of the secondpixels included in the cavity part of each of the spheroids. Note thatin the refocusing processing for all the pixels in the three-dimensionalspace, the internal image generation unit 122 may generate the in-focusimages on all focal planes. The discrimination unit 123 may extract theouter shape of each of the spheroids and the cavity part of each of thespheroids in the in-focus image of each focal plane.

Second Embodiment

A culture state determination device 20 according to the secondembodiment will be described. The culture state determination device 20according to the second embodiment calculates the size of the pluralityof the spheroid regions in the reference captured image. Furthermore, ifthe distribution in the size of the plurality of the spheroid regions islarge, the culture state determination device determines that theculture state is bad. Hereinafter, the second embodiment will bedescribed with a focus on differences from the first embodiment.

2-1. Configuration of Culture State Determination Device According toSecond Embodiment

FIG. 21 is a block diagram illustrating one example of a functionalconfiguration of the culture state determination device 20 according tothe second embodiment. In FIG. 21 , substantially the same components asthose in FIG. 1 are denoted by the same reference numerals, anddescription thereof will be omitted appropriately. As illustrated inFIG. 21 , the culture state determination device 20 comprises theimaging device 100, the storage unit 110, an image processing unit 220,a calculation unit 230, a state determination unit 240, and the displayunit 150. Further, the image processing unit 220 comprises a regionextraction unit 221, the internal image generation unit 122, and thediscrimination unit 123. The configurations of the imaging device 100and the storage unit 110 are the same as those in the first embodiment.

Similarly to the region extraction unit 221 according to the firstembodiment, the region extraction unit 221 of the image processing unit220 determines a reference captured image from a plurality of thecaptured images, and extracts regions where a spheroid image is present,namely, the spheroid regions, from the reference captured image.Furthermore, the region extraction unit 221 assigns an ID to each of theextracted spheroid regions. The region extraction unit 221 storesinformation such as the ID and position of each extracted spheroidregion in the storage unit 110 in association with the referencecaptured image from which the spheroid regions have been extracted.

In the present embodiment, the region extraction unit 221 calculatespixel coordinates of the pixels forming the spheroid regions on thereference captured image as information on the spheroid regions.Furthermore, the region extraction unit 221 assigns, to the calculatedpixels, the same ID as the spheroid regions formed by the pixels. Forexample, FIG. 22 schematically illustrates one example of spheroidregions extracted from the reference captured image. The squares in FIG.22 schematically show a part of the pixels of the reference capturedimage. The region extraction unit 221 extracts five spheroid regions A1to A5 in the part of the reference captured image shown in FIG. 22 , andassigns 001 to 005 as IDs to the five spheroid regions A1 to A5,respectively. Further, the same IDs 001 to 005 as those of the spheroidregions are assigned to the pixels included in the spheroid regions A1to A5 as labels. The region extraction unit 221 stores the pixelcoordinates on the reference captured image in the pixels included ineach of the spheroid areas A1 to A5 and the ID001 to 005 of the pixelsin the storage unit 110 in association with each other. In the exampleof FIG. 23 , the top left vertex on the drawing, which is one of thevertices of the reference captured image, is defined as the origin ofthe pixel coordinates, the x coordinate is defined from the origin tothe left, and the y coordinate is defined from the origin to the bottom.

The region extraction unit 221 may incorporate information on the pixelcoordinates and ID of the spheroid region into a file of the referencecaptured image as shown in FIG. 22 . In this case, the information onthe spheroid region is stored in the storage unit 110 as an image file.Alternatively, the region extraction unit 221 may generate data of thepixel coordinates and the ID of the spheroid region and store the datain the storage unit 110 so as to form a table as shown in FIG. 23 . FIG.23 shows one example of the content stored in the storage unit 110 forthe information on the spheroid region.

The configurations of the internal image generation unit 122 and thediscrimination unit 123 of the image processing unit 220 are the same asthose in the first embodiment.

The calculation unit 230 extracts the information on the spheroidregions extracted by the region extraction unit 221 and stored in thestorage unit 110, and calculates the size of each spheroid region.Specifically, the calculation unit 230 sets, for each spheroid regionstored as a continuous pixel region, a minimum ellipse including thespheroid region on the image coordinates of the reference capturedimage, and calculates the major axis and minor axis of the ellipse. Notethat the image for setting the ellipse may be a captured image otherthan the reference captured image. For example, as shown in FIG. 22 ,the calculation unit 230 sets the minimum ellipses C1 to C5circumscribing the spheroid regions A1 to A5, respectively, andcalculates the major axis and minor axis of each of the ellipses C1 toC5. The ellipse may include a circle and an ellipse. Furthermore, thecalculation unit 230 calculates the sum of the lengths of the major axisand minor axis of each ellipse, and determines the sum as the size ofthe spheroid. The calculation unit 230 may calculate a size distributionbased on the ellipses of all spheroid regions, for example, a statisticsuch as a maximum value, a minimum value, a median value, an average,and a variance. Furthermore, the calculation unit 230 may create ahistogram indicating the distribution of the size of all spheroidregions. The size of the spheroid region is not limited to the sum ofthe lengths of the major axis and minor axis of the smallest ellipseincluding the spheroid region. For example, the size of the spheroidregion may be the area of the region, the total area of the pixelsincluded in the region, namely, the number of the pixels, the area ofthe smallest polygon including the spheroid region, or the sum of thelengths of diagonals of the smallest polygon.

Further, the calculation unit 230 calculates the first ratio between thefirst total number and the second total number that are the total numberof the first and second pixels in the entire spheroid region,respectively, on the basis of the number of the first pixels in eachspheroid and the number of the second pixels in the cavity part in thespheroid, both of which have been discriminated by the discriminationunit 123

The state determination part 240 determines the quality of the culturestate using the information on the distribution of the size of thespheroids calculated by the calculation unit 230. The statedetermination part 240 determines that the culture state is bad, if thedistribution of the size of the spheroids is large, and determines thatthe culture state is good, if the distribution of the size of thespheroid is small. The state determination part 240 determines thedistributions by applying a reference of the distribution, for example,a predetermined fourth threshold value, to the distribution of the sizeof the spheroids.

The fourth threshold value is a threshold value indicating that theculture state is bad if the distribution of the size of the spheroids isgreater than or equal to the fourth threshold value, and that theculture state is not bad if the distribution of the size of thespheroids is less than the fourth threshold value. The fourth thresholdvalue can be determined to have various values depending on conditionssuch as the kind and the quantity of cells forming the spheroid, thetime of the culture of the spheroid, the state of the required qualityof the spheroid, and the use of the spheroid. For example, the fourththreshold value may be determined on the basis of a statistical resultof a relationship between the distribution of the sizes of the pluralityof the spheroids detected by experiments and the culture state of thespheroids. Such a fourth threshold value may be determined depending onthe above conditions by a designer, a manufacturer, or a user of theculture state determination device 10, and the determined fourththreshold value may be input via an input device (not shown) and storedin the storage unit 110.

Furthermore, the state determination part 240 compares the first ratiobetween the first total number and the second total number to apredetermined determination reference value, and determines that theculture state is good if the first ratio is lower than the determinationreference value, and that the culture state is bad if the first ratio isgreater than or equal to the determination reference value.

The display unit 150 shows the distribution of the size of the spheroidscalculated by the calculation unit 230 and the result determined by thestate determination part 240. The display unit 150 may display thedisplay content on a display, for example, as a graph, characters,symbols, images, etc., may be displayed as a sound or an acoustic signalon a speaker, or may be displayed in another display method.

2-2, Operation of Culture State Determination Device According to SecondEmbodiment

The operation of the culture state determination device 20 according tothe second embodiment will be described with reference to FIG. 24 . FIG.24 is a flowchart showing one example of the operation of the culturestate determination device 20 according to the second embodiment. InFIG. 24 , substantially the same steps as those in FIG. 11 are denotedby the same reference numerals, and description thereof will be omittedas appropriate.

First, in the step S1100, the imaging device 100 performs the sameprocessing as the step S1100 of the first embodiment. Next, in the stepS2200, the region extraction unit 221 of the image processing unit 220determines a reference captured image from the captured image acquiredin the step S1100, and extracts a spheroid region from the referencecaptured image. For each of the one or more spheroid regions extractedfrom the reference captured image, the region extraction unit 221 storespixel coordinates on the reference captured image of the pixels includedin the region and an ID of the region that is the label of the pixel inassociation each other in the storage unit 110.

Next, in the step S2300, the calculation unit 230 determines the size ofeach of the spheroid regions on the reference captured image on thebasis of the information on the pixel coordinates and IDs of the pixelsincluded in the spheroid regions extracted in the step S2200 and storedin the storage unit 110. Furthermore, the calculation unit 230 storesinformation on the size of each spheroid region in the storage unit 110in association with the ID of the region. In the present embodiment, thesize of the spheroid region is, for example, the sum of the lengths ofthe major axis and minor axis of the smallest ellipse including theregion. In the present embodiment, the index of the size of the spheroidregion is the sum of the lengths of the major axis and minor axis of theellipse; however, the index may be another index such as length ofdiagonal in a polygon such as the smallest rectangular polygon includingthe spheroid region, the sum of the diagonals of the polygon, the numberof the pixels included in the spheroid area, or the square root thereof.The calculation unit 230 may create a histogram indicating thedistribution of the size of all spheroid regions based on the size ofeach spheroid region and store the histogram in the storage unit 110.The calculation unit 230 may calculate a statistic amount of the size ofthe spheroids region based on the size of each spheroid region and storethe statistic amount in the storage unit 110.

Next, in the step S2400, based on the index of the size of the spheroidregion calculated in the step S2300, namely, the sum of the lengths ofthe major axis and minor axis of the smallest ellipse including thespheroid region, the state determination part 240 determines whether ornot the distribution of the size of all the spheroid regions extractedin the step S2200 is large. At this time, the state determination part240 determines whether or not the distribution of the size of allspheroid regions is larger than a predetermined variation reference. Forexample, the state determination part 240 calculates the distribution ofthe size of all spheroid regions, and determines whether or not thedistribution is larger than the fourth threshold value. If thedistribution of the size of the spheroids region is greater than orequal to the reference of the distribution, namely, if the distributionof the size of all the spheroid regions is equal to or greater than thefourth threshold value (Yes in the step S2400), the state determinationpart 240 proceeds to the step S2500. If the distribution of the size ofthe spheroid regions is smaller than the reference of the distribution,namely, if the distribution of the size of all spheroid regions is lessthan the fourth threshold value (No in the step S2400), the statedetermination part 240 proceeds to the step S1300.

Next, in the step S2500, the state determination part 240 determinesthat the culture state of the culture vessel containing the spheroiddetermined that the distribution of the size is large in the step S2400is bad. In other words, the state determination part 240 determines thatthe culture state of the entire one culture vessel is bad. Furthermore,the state determination part 240 determines to discard all the spheroidsof the culture vessel. In other words, the state determination part 240determines the discard of the spheroids for each culture vessel. Thestate determination part 240 proceeds to the step S2700 after theprocessing of the step S2500.

The processing in the steps S1300 to S1600 is the same as that in thefirst embodiment. After the processing in the step S1600, the statedetermination part 240 proceeds to the step S2700.

In the step S2700, the display unit 150 displays the distribution of thesize of the spheroids calculated in the step S2300 and determined in thestep S2400 on the display. For example, the display unit 150 displaysthe size of the spheroid region calculated in the step S2300 as the sizeof the spheroid. Furthermore, the display unit 150 displays a histogramindicating the distribution of the size of the plurality of the spheroidregions as the distribution of the size of the spheroids. The displayunit 150 may display a statistic amount such as the minimum value, themaximum value, the variance, and the standard deviation of the size ofthe plurality of the spheroid regions on the display. Further, thedisplay unit 150 also displays the determination result of the stepS1600. The display unit 150 may display simultaneously the display ofthe distribution of the size of the spheroids and the display of thedetermination result of the culture state, or may switch and display oneof them. The display unit 150 may display the above information togetherwith the display by an image, or separately from the display by theimage, by outputting an audio signal.

For example, FIGS. 25A to 25D show examples of the display by thedisplay unit 150. In this example, the display unit 150 is a display fordisplaying an image, and FIGS. 25A to 25D schematically show theexamples of display screens on the display of the display unit 150.

FIG. 25A shows one state of a display screen 150 a of the display unit150. FIG. 25A shows a case where the culture state determination device20 determines that the culture state of the spheroid is bad based on thedistribution of the size of the spheroids. In the display screen 150 a,an image 150 b of the entire culture vessel is displayed in the leftregion. In the image 150 b, the spheroids extracted as the spheroidregions in the step S2200 are displayed. Each spheroid is indicated by aline in which the outline of the spheroid region is emphasized, or by acircle or an ellipse surrounding the spheroid region. The image 150 b isa selected one of the plurality of the captured images acquired in thestep S1100, and may be a reference captured image. For example, thereference captured image is an image captured during the illumination bythe illuminator 101 located immediately above the center point of theimage sensor 102.

In the upper right region of the display screen 150 a, a histogram 150 cindicating the distribution of the size of the spheroids calculated inthe step S2300 is displayed. Further, in the display screen 150 a,statistical information 150 d on the spheroids is displayed in the lowerright region. The statistical information 150 d includes the number ofthe extracted spheroid regions, namely, the number of the spheroids andinformation on the size of the spheroids. In this example, theinformation on the size of the spheroids is the average, maximum value,minimum value, and variance of the size of the spheroids; however, isnot limited thereto. Further, the statistical information 150 dindicates the relationship between the variance and the fourth thresholdvalue yy. In this example, it is indicated that the dispersion is largerthan the fourth threshold value, namely, the dispersion is not less thanthe fourth threshold value. Thereby, the user who has seen theinformation 150 d can recognize that the distribution of the size of thespheroids is large and that the spheroids are not suitable for theprocessing after the culture. In this case, the culture statedetermination device 20 determines that the distribution of the size ofthe spheroids is large in the step S2400, and does not perform theprocessing from the step S1300 to the step S1600. Therefore, since thereis no information other than the information displayed in FIG. 25A,there is no display, on the display screen 150 a, that presents callingof other information or switching of the display screen.

FIG. 25B shows another state of the display screen 150 a of the displayunit 150. FIG. 25B shows a case where the culture state determinationdevice 20 determines that the culture state of the spheroids is not badbased on the distribution of the size of the spheroids. In the displayscreen 150 a, similarly to the case of FIG. 25A, the image 150 b of theentire culture vessel, the histogram 150 c showing the distribution ofthe size of the spheroids, and the statistical information 150 d on thespheroids are displayed. In this case, a “cell amount display” icon 150e for displaying other information is displayed in the lower left regionof the display screen 150 a, namely, below the image 150 b. Further, inthe statistical information 150 d, there is no display indicating thatthe distribution of the size of the spheroids is equal to or greaterthan the fourth threshold value. In this case, the culture statedetermination device 20 determines that the distribution of the size ofthe spheroids is within the reference in the step S2400, performs therefocusing processing through the processing from the step S1300 to thestep S1600, and generates the in-focus image of each spheroid region.Then, the culture state determination device 20 calculates the firstratio between the total number of the first pixels in the outer shape ofthe spheroid and the total number of the second pixels in the cavitypart of the spheroid.

FIG. 25C shows another state of FIG. 25B on the display screen 150 a ofthe display unit 150. The display screen 150 a in FIG. 25C is oneexample of a screen that is displayed after an input operation such asclicking on the “cell amount display” icon 150 e in FIG. 25B. In thedisplay screen 150 a, similarly to the case of FIG. 25B, the image 150 bof the entire culture vessel, the histogram 150 c indicating thedistribution of the size of the spheroids, and the statisticalinformation 150 d on the spheroids are displayed. Further, an input unit150 f for designating a focal plane to be displayed on the image 150 bis displayed adjacent to the image 150 b. In this example, the inputunit 150 f is a slider. If the user moves the slider of the input unit150 f on the display screen 150 a, the display unit 150 displays anin-focus image of each spheroid on the focal plane at a positioncorresponding to the position of the slider as the image 150 b. In otherwords, the display unit 150 can display a cross-sectional image of eachspheroid at an arbitrary focal plane. The moving direction of the slidercorresponds to a direction toward and away from the light receivingsurface of the image sensor 102, and the position of the slidercorresponds to a distance from the light receiving surface of the imagesensor 102.

Furthermore, information 150 g on the cavity part of the spheroid isdisplayed in the lower left region of the display screen 150 a, namely,below the image 150 b. The information 150 g includes the total numberof the first pixels in the outer shape of the spheroid, the total numberof the second pixels in the cavity part of the spheroid, and a cavitypart ratio. The cavity part ratio is the first ratio. In this example,it is displayed that the cavity part ratio exceeds the determinationreference value. As a result, the user who has seen the information 150g can recognize that the ratio of the cavity part of the spheroid islarge, and the spheroid is not suitable for the processing after theculture.

Further, the input unit 150 f may have any configuration other than theslider, as long as the focal plane can be selected through the inputunit 150 f. For example, the input unit 150 f may be a key for receivingan input of a parameter such as a numerical value indicating theposition of the focal plane, a touch panel for changing the focal planeto be displayed by receiving an input such as a slide on the image 150b, or a pointing device for selecting the focal plane.

FIG. 25D shows another state of FIG. 25C on the display screen 150 a ofthe display unit 150. The display screen 150 a in FIG. 25D is oneexample of a screen that displays an enlarged image 150 h of a specificspheroid in a case where an image of the specific spheroid is designatedon the display screen 150 a in FIG. 25C. The designation of the image ofthe specific spheroid may be performed using a pointing device such as acursor or a pointer in the display screen 150 a. The enlarged image 150h may include an input unit 150 ha for selecting a focal plane on whichthe in-focus image of the spheroid is displayed. The input unit 150 hamay have the same configuration as the input unit 150 f, and is a sliderin this example. As a result, the user can display and visuallyrecognize the arbitrary cross-sectional image of the selected spheroidon the enlarged image 150 h.

In this way, the culture state determination device 20 uses the displayscreens 150 a as shown in FIGS. 25A to 25D not only to display thedetermination results of the culture state for each culture vessel basedon the distribution of the size of the spheroids and the amount of thecells in the spheroid but also to display a three-dimensional image ofindividual spheroids and provide a user with detailed information on thespheroids.

2-3. Effect

As described above, first, the culture state determination device 20according to the second embodiment determines the distribution of thesize of the plurality of the spheroids for the plurality of thespheroids cultured in the culture vessel. The culture statedetermination device 20 determines that all the spheroids in the culturevessel are discarded if the distribution is larger than the reference,and further determines the culture state if the distribution is smallerthan the reference. As a result, when desired cells are acquired by theprocessing after the culture such as a differentiation processing, it ispossible to easily select an efficient culture vessel. In the furtherdetermination of the culture state, with regard to the plurality of thespheroids in the culture vessel that have not been discarded due to thedistribution, the culture state determination device 20 determines thequality of the culture state in the entire culture vessel based on thefirst ratio of the volume of the cavity part to the volume of thespheroid in the entire culture vessel, similarly to the case of thefirst embodiment.

The processing amount for calculating the first ratio with therefocusing processing is relatively large. On the other hand, theprocessing amount for determining the distribution of the size of theplurality of the spheroids in the culture vessel is significantlysmaller than the processing amount for calculating the first ratio. Theculture state determination device 20 improves processing speed fordetermining the culture state of the spheroids in the plurality of theculture vessels by decreasing the number of the culture vessels forwhich the first ratio is to be calculated based on the distribution ofthe size of the spheroids. As described above, the culture statedetermination device 20 determines the culture state of the spheroidsduring the culture for each culture vessel, and enables cultured cellsin a state suitable for differentiation processing to be extractedefficiently.

Other Embodiments

Although the culture state determination device according to one or moreaspects has been described based on the embodiments, the presentdisclosure is not limited to these embodiments. Various modificationsconceived by those skilled in the art and forms constructed by combiningcomponents in different embodiments are also within the scope of the oneor more aspects, unless deviating from the gist of the presentdisclosure.

The culture state determination device according to the embodimentcalculates, for each spheroid, the number of the first pixels in theouter shape of the spheroid and the number of the second pixels in thecavity part of the spheroid on the in-focus image of each focal plane.Furthermore, the culture state determination device calculates the firsttotal number by calculating the sum of the number of the first pixels atall the focal planes of all the spheroids, and calculates the secondtotal number by calculating the sum of the number of the second pixelsat all the focal planes of all the spheroids. However, the calculationmethod of the first total number and the second total number is notlimited thereto. For example, for each spheroid, the culture statedetermination device may calculate the first total number by calculatingthe sum of the numbers of the first pixels at all focal planes andcalculating the sum of the numbers of the first pixels of all thespheroids. Similarly, for each spheroid, the culture state determinationdevice may calculate the second total number by calculating the sum ofthe numbers of the second pixels at all focal planes and calculating thesum of the numbers of the second pixels of all the spheroids. In thiscase, the culture state of each spheroid can be determined bycalculating the volume of each spheroid and the volume of the cavitypart of each spheroid.

Further, as described above, the technology of the present disclosuremay be realized by a system, a device, a method, an integrated circuit,a computer program, or a recording medium such as a computer-readablerecording disk. The the technology of the present disclosure may berealized by any combination of the system, the device, the method, theintegrated circuit, and the computer program and the recording medium.The computer-readable recording medium includes a non-volatile recordingmedium such as a CD-ROM.

For example, each processing unit included in the culture statedetermination device according to the above embodiment is typicallyrealized as a large scale integration (i.e., LSI) that is an integratedcircuit. These may be individually made into one chip, or may be madeinto one chip so as to include a part or all of them.

Further, the circuit integration is not limited to LSI, and may berealized by a dedicated circuit or a general-purpose processor. Afieldprogrammable gate array (i.e., FPGA) that can be programmed aftermanufacturing the LSI, or a reconfigurable processor that canreconfigure the connection and setting of circuit cells inside the LSImay be used.

In the above embodiment, each component may be configured by dedicatedhardware or may be realized by executing a software program suitable foreach component. Each component may be realized by a program executionunit such as a processor such as a CPU reading and executing a softwareprogram recorded on a recording medium such as a hard disk or asemiconductor memory.

In addition, some or all of the above-described components may beconfigured from a removable integrated circuit (i.e., IC) card or asingle module. The IC card or the module is a computer system thatincludes a microprocessor, ROM, and RAM. The IC card or the module mayinclude the above-described LSI or a system LSI. The IC card or themodule achieves its function by the microprocessor operating inaccordance with the computer program. These IC cards and modules mayhave tamper resistance.

The culture state determination method of the present disclosure may berealized by a circuit such as a micro processing unit (i.e., MPU), aCPU, a processor, an LSI, an IC card, or a single module.

Furthermore, the technology of the present disclosure may be realized bya software program or a digital signal consisting of a software program,and may be a non-transitory computer-readable recording medium on whichthe program is recorded. In addition, the program can be distributed viaa transmission medium such as the Internet.

In addition, all the numbers such as the ordinal numbers and the amountsused above are exemplified for specifically explaining the technology ofthe present disclosure, and the present disclosure is not limited to theillustrated numbers. In addition, the connection relationship betweenthe constituent elements is exemplified for specifically explaining thetechnology of the present disclosure, and the connection relationshipfor realizing the functions of the present disclosure is not limitedthereto.

In addition, division of functional blocks in the block diagram is oneexample. A plurality of functional blocks may be realized as onefunctional block. One functional block may be divided into a pluralityof parts, or some functions may be transferred to other functionalblocks. In addition, functions of a plurality of functional blockshaving similar functions may be processed in parallel or time-divisionby a single hardware or software.

INDUSTRIAL APPLICABILITY

The technique of the present disclosure can be widely used for atechnique for determining the culture state of stem cells such as tissuestem cells, iPS cells and ES cells in culture, or cell aggregations suchas embryos. The technique of the present disclosure is useful fordetermining whether or not the culture state is suitable fordifferentiation processing when spheroids of pluripotent cells such asthe stem cells are cultured and subjected to the differentiationprocessing.

REFERENCE SIGNS LIST

-   10, 20 Culture state determination device-   100 Imaging device-   101, 101 a, 101 b Illuminator(s)-   102 Image sensor-   103 Imaging control unit-   110 Storage unit-   120, 220 Image processing unit-   121, 221 Region extraction unit-   122 Internal image generation unit-   123 Discrimination unit-   130, 230 Calculation unit-   140, 240 State determination unit-   150 Display unit-   1221 Refocusing Unit-   1222 Focal plane table-   1223 Image generation unit

The invention claimed is:
 1. A culture state determination device,comprising: a plurality of light sources; an image sensor on which aplurality of cell aggregations are to be mounted; and control circuitrywhich, in operation, (a) repeatedly causes the image sensor to acquirean image of the plurality of cell aggregations when the plurality ofcell aggregations are illuminated with each of the plurality of lightsources sequentially, to acquire a plurality of images, wherein each ofthe plurality of images includes an image of the plurality of cellaggregations, (b) extracts an image region including an image of onecell aggregation from each of the plurality of images; (c) generatesthree-dimensional image information with regard to the image regionusing the plurality of images; (d) calculates a first volume and asecond volume from the three-dimensional image information, wherein: thefirst volume is an entire volume of the one cell aggregation, and thesecond volume is a volume of a cavity part of the one cell aggregation;(e) repeats the steps (b)-(d) to obtain first volumes of the pluralityof cell aggregations, respectively, and second volumes of the pluralityof ell aggregations, respectively; (f) calculates a first total volumeand a second total volume, wherein: the first total volume is a sum ofthe first volumes of the plurality of cell aggregations, and the secondtotal volume is a sum of the second volumes of the plurality of cellaggregations; and (g) determines a culture state of the plurality ofcell aggregations using the first total volume and the second totalvolume, wherein: the control circuitry determines whether or notdistribution of size of regions of the plurality of cell aggregations islarger than a predetermined variation reference, after the step (b) andbefore the step (c), and if the control circuitry determines that thedistribution of the size of regions is larger than the predeterminedvariation reference, the control circuitry determines that the culturestate of the plurality of cell aggregations is bad without performingthe steps (c) to (f).
 2. A method for determining a culture state, themethod comprising: (a) repeatedly causing an image sensor to acquire animage including a plurality of cell aggregations when the plurality ofthe cell aggregations are illuminated with each of a plurality of lightsources sequentially, to acquire a plurality of images, wherein each ofthe plurality of images includes an image of the plurality of cellaggregations; (b) extracting an image region including an image of onecell aggregation from each of the plurality of images; (c) generatingthree-dimensional image information with regard to the image regionusing the plurality of images; (d) calculating a first volume and asecond volume from the three-dimensional image information, wherein: thefirst volume is an entire volume of the one cell aggregation, and thesecond volume is a volume of a cavity part of the one cell aggregation;(e) repeating the steps (b)-(d) to obtain first volumes of the pluralityof cell aggregations, respectively, and second volumes of the pluralityof ell aggregations, respectively; (f) calculating a first total volumeand a second total volume, wherein: the first total volume is a sum ofthe first volumes of the plurality of cell aggregations, and the secondtotal volume is a sum of the second volumes of the plurality of cellaggregations; (g) determining a culture state of the plurality of cellaggregations using the first total volume and the second total volume;determining whether or not distribution of size of regions of theplurality of cell aggregations is larger than a predetermined variationreference, after the step (b) and before the step (c); and afterdetermining that the distribution of the size of regions is larger thanthe predetermined variation reference, determining that the culturestate of the plurality of cell aggregations is bad without performingthe steps (c) to (f).
 3. A culture state determination device,comprising: a plurality of light sources; an image sensor on which aplurality of cell aggregations are to be mounted; and control circuitrywhich, in operation, (a) repeatedly causes the image sensor to acquirean image of the plurality of cell aggregations when the plurality ofcell aggregations are illuminated with each of the plurality of lightsources sequentially, to acquire a plurality of images, wherein each ofthe plurality of images includes an image of the plurality of cellaggregations, (b) extracts an image region including the image of theplurality of cell aggregations from each of the plurality of images; (c)generates three-dimensional image information with regard to the imageregion using the plurality of images; (d) calculates a first volume anda second volume from the three-dimensional image information, wherein:the first volume is an entire volume of each of the plurality of cellaggregations, and the second volume is a volume of a cavity part of eachof the plurality of cell aggregations; (e) calculates a first totalvolume and a second total volume, wherein: the first total volume is asum of first volumes with respect to the plurality of cell aggregations,and the second total volume is a sum of second volumes with respect tothe plurality of cell aggregations; and (f) determines a culture stateof the cell aggregation using the first total volume and the secondtotal volume, wherein: the control circuitry determines whether or notdistribution of size of regions of the plurality of cell aggregations islarger than a predetermined variation reference, after the step (b) andbefore the step (c), and if the control circuitry determines that thedistribution of the size of regions is larger than the predeterminedvariation reference, the control circuitry determines that the culturestate of the plurality of cell aggregations is bad without performingthe steps (c) to (f).
 4. A method for determining a culture state, themethod comprising: (a) repeatedly causing an image sensor to acquire animage of a plurality of cell aggregations when the plurality of cellaggregations are illuminated with each of a plurality of light sourcessequentially, to acquire a plurality of captured images, wherein each ofthe plurality of images includes an image of the plurality of cellaggregations, (b) extracting an image region including the image of thecell aggregation from each of the plurality of images; (c) generatingthree-dimensional image information with regard to the image regionusing the plurality of images; (d) calculating a first volume and asecond volume from the three-dimensional image information, wherein: thefirst volume is an entire volume of each of the plurality of cellaggregations, and the second volume is a volume of a cavity part of eachof the plurality of cell aggregations; (e) calculating a first totalvolume and a second total volume, wherein: the first total volume is asum of first volumes with respect to the plurality of cell aggregations,and the second total volume is a sum of second volumes with respect tothe plurality of cell aggregations; (f) determining a culture state ofthe plurality of cell aggregations using the first total volume and thesecond total volume; determining whether or not distribution of size ofregions of the plurality of cell aggregations is larger than apredetermined variation reference, after the step (b) and before thestep (c); and after determining that the distribution of the size ofregions is larger than the predetermined variation reference,determining that the culture state of the plurality of cell aggregationsis bad without performing the steps (c) to (f).